Scripting a HEX edit for unrecognized RAW files

It happens every once in a while. Someone buys a new camera with awesome bells and whistles, snaps away some shots and then as they start their workflow, they hit a file format wall. Often the format of the file is exactly the same as older cameras of the line, with a change to the camera type field. RAW converters use that field to determine settings for color profile, white balance, interpolation type etc.

So when your new Canon T4i isn't "recognized", but the application knew what to set for T3i and T2i cameras which used the same sensor, and family of image processors, you really wonder why there isn't an automatic setting you can simply configure to treat the RAW file like that of it's predecessors. It's something that makes sense, and more so, why can't you just download a text or XML configuration for the camera and import it. I'm looking at you Adobe...

A perfect world where software makes logical sense for the user doesn't exist. Logical sense for a commercial product mean getting users to update and spend money so perhaps that's a reason why importing a profile doesn't exist.

So let's get down to understanding what we can do.

1. A previous model of camera is properly understood
2. The name is what many applications use to recognize the format and apply profiles
3. The profile would be almost identical for a new model of camera
4. File format difference is minimal - a couple bytes that make up the name normally

Logically Lets just change those bytes!

NB: there are command line tools that will do this in a simpler way. I just choose to script because it's more satisfying ;)

To change the bytes of the RAW binary file, we can convert to HEX, perform a substitution, and reconvert to binary.

To convert a file to HEX, there's "xxd". It's a unix command, but also available on windows, through installing GVIM.

Tools needed:
SED : https://github.com/bmatzelle/gow/wiki
XXD : http://www.vim.org/download.php

We will make this easy to use in 3 steps:

• Install tools SED and VIM (xxd)
• Create BAT file
• Put BAT file in "send to" folder for easy right click  menu

E.g. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Looking a a small random file "subnip.au3"
C:\temp>type subnip.au3
MsgBox(0,0,BitAND(248,108))
C:\temp>"c:\Program Files\Vim\vim73\xxd.exe" subnip.au3
0000000: 4d73 6742 6f78 2830 2c30 2c42 6974 414e  MsgBox(0,0,BitAN
0000010: 4428 3234 382c 3130 3829 29              D(248,108))

Great, that worked. We can use this format to identify what we want changed. Use the column on the right to easily read the text to change, and then determine what hex code to change.

For example, lets say I wanted to change 248,108 to 249,110.
The 2 starts on the third character of that line:
D(248,108))
With the hex code, every 2 characters is one ASCII character, so your match is here:
4428 3234 382c 3130 3829 29
So matching up what we want changed:
D(248,108)) 
4428 3234 382c 3130 3829 29
We'd need to rplace the highlighted areas with the text we want.

The UNIX tool "sed" is perfect for this.
first, let's simplify our xxd output.

C:\temp>"c:\Program Files\Vim\vim74\xxd.exe" -ps subnip.au3
4d7367426f7828302c302c426974414e44283234382c3130382929

Perfect - now it's easier to use sed. Let's match what we want:
4428 3234 382c 3130 3829 29
4d7367426f7828302c302c426974414e44283234382c3130382929
From this, the string we want to change is
44283234382c3130382929 to
44283234392c3131302929 
i.e. we've changed to HEX representation of the ascii codes we want to use.
NB: This is a byte for byte replacement. We're not adding bytes, or removing bytes.


Lets put this all together!
C:\temp>"c:\Program Files\Vim\vim74\xxd.exe" -ps subnip.au3 | sed "s/44283234382c3130382929/44283234392c3131302929/" | "c:\Program Files\Vim\vim74\xxd.exe" -ps -r

MsgBox(0,0,BitAND(249,110))

Huzzah! Big success! The same technique can be applied to a larger RAW file to fix our camera support.


 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=

Now onto RAW HEX editing.
I was able to get RAW files from here:
http://www.imaging-resource.com/PRODS/olympus-e-m10/olympus-e-m10A7.HTM

In this script, I will take Olympus EM10 RAW, and change the camera type to EM5. EM10 is the new model, which is almost exactly the same sensor and processing as the EM5.

Start by running xxd against RAWs and dumping them to text files. Open in VIM and compare. Look for where the Camera name is kept, and what to change.

From examining HEX dumps in VIM, we can determine the strings to replace.
In converting EM10 to EM5, we're converting
452d4d3130202020202020202020 to
452d4d3520202020202020202020

Let's make a script to take the incoming filename as a parameter and simplify the process.

::+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

:: go to "%appdata%\Microsoft\Windows\SendTo" in file explorer
:: drop this bat file in there

set xxdbin="c:\Program Files (x86)\Vim\vim74\xxd.exe"
set srt=%1
set fin=%1.em10conv.orf
echo converting %srt% to %fin%

set iem10=452d4d3130202020202020202020
set iem5=452d4d3520202020202020202020

%xxdbin% -ps -c 4096 %srt% |sed "s/%iem10%/%iem5%/"|%xxdbin% -ps -c 4096 -r > %fin%

::+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Perfect - lets make it easy!
Navigate to "%appdata%\Microsoft\Windows\SendTo" in file explorer, and save the BAT file here.

After those 3 steps - install GVIM, GOW and bat file in "send to" -
Now when you right click on a file, you can "send to" this bat file.
The output will appear in the same folder with same name as the original+"em10conv.orf"

Overall the process isn't that bad really. It's just a several little things to put together.

A little test of 4 different 25mm F1.4 lenses on Micro Four Thirds

So, I'm not fond of formally testing lenses myself (though I do like to read em). Comparing a couple lenses may be fairly quick (quick and dirty may not check everything ☺), but putting four lenses into an easily readable report takes up some time. So I'm not doing this with too much depth. This is just to assert to myself the feel of these different solutions, that share the same 25mm f1.4 specification, and look at more obvious differences in their results.

This is a very different kind of roundup. There's only one lens here I'd expect people to come across "in the wild". The next most common is a C-mount so yes, lets get weird...

The lenses here (on my Olympus EP3) are:

1. Panasonic (Leica) DG Summilux 25mm F1.4 ASPH.
2. Pentax TV lens 25mm F1.4 (1" coverage)
3. Wollensak Cine Raptor 25mm F1.5
4. Nikon 35mm F2.0 AI Nippon Kogaku

First off, some of these names just simply sound exotic. It brings to mind images of awesomeness that simple named lenses can't match. Obviously, fancy names equal good image quality...
Sarcasm aside, I like these names. It really helps set these aside from just being a mundane amalgam of specifications.





About the lenses:

Summilux is what Leica calls their current batch of lenses with an F1.4 aperture. Leica itself is a luxury brand in cameras. Leica's been at this for a while - their compact landscape camera dates back to 1913. They've got history, and a name that carries weight, so it makes sense that Panasonic would want to ride that name. The lens is Panasonic made but designed or approved by Leica. This is a current production lens for the high grade MFT market.

The most pedestrian of names is the Pentax TV lens. Quality is a bit better than the cheap 25mm c-mounts lenses on ebay that go for $25-$50, but it's not that great. It is quite comfortable to use with it's smooth ring though. This lens still exists, but is now branded Ricoh. This is a current low cost CCTV lens. Stepless aperture is good for video. While Pentax is also steeped in optical history starting with spectacle lenses, a mass production and cheaper lens like this probably inherits little of their mojo.

Wollensak was making camera lenses as far back as 1902,and shutters from 1899. They pushed high speed photography and video with their fastax brand, developing on a Bell Labs spinning prism implementation taking frame rates up to 18,000fps in the 1960s. For their film cameras, they have a line of cine lenses (c-mount)that were good value at the time. The company went under in 1972, so the lens here is at least as old as that.
 
Nikon has been around since 1917, but at the time was known as Nippon Kōgaku Kōgyō Kabushikigaisha (say that ten times fast...). It stayed that way for a long time, but in 1988, someone thought that it was too much of a mouthful, so the company was renamed to Nikon Corporation. "Nikon" dates back to 1946 and was formed from the words "Nippon" and "Ikon". Ikon was a Zeiss brand. In 1959, Nikon started pushing their Nikkor F line of SLR cameras. The 35mm F2.0 in use is an optical design introduced by Nikon in 1962. In 1975 Nikon updated the lens to stop down to F22, and since my lens doesn't have that option, the age is between 1962 and 1975 - possibly closer to 1975 since Nikon initially used chrome noses instead of all black on their lenses.

The Panasonic Leica produces a beautiful rendering. It's sharp to the edge, and the bokeh is smooth. Focus is quick. Manual focus is horrible since the focus ring is electronically connected to the glass and not mechanical. however, this gives it a relatively compact size for this level of quality and speed. It should be noted - this lens would be auto corrected in the camera for certain traits (CA/distortion etc), so the JPGs used would favor this more than other lenses here.

Ye olde Nikon 35mm F2.0 on focal reducer. Slight difference in focal length. Quality is decent, but it's definitely not as sharp. I can't say it's purely the focal reducer or the lens, the fault is probably shared between the two.

That said, it's sharp enough to satisfy most and make a good picture. Bokeh is nice, maybe the nicest of the bunch. it's focused a few millimeters in front of the Panasonic, so blurring is a little different, but as you can see it's quite smooth.

Also of note - the entire image shows little or no light falloff. Even on a full frame camera, this lens was well known to deliver low vignetting.

I loved using this Wollensak lens. Before MFT had a lot of lenses available, people were adapting any fast prime they could find to get a decent F-Stop. Classic manual focus 50mm lenses were common, and cheap, but they were also an equivalent 100mm on this format. The field of view more specific in usage compared to a "normal". C-mount lenses like these 25mm lenses were perfect fits for the system - small, light, and some actually delivered decent quality.

The Wollensak here shows a very sharp center quality - surprisingly sharper than the 35mm F2.0 on focal reducer. while it's not quite F1.4, the F1.5 is close - however it was never meant to cover this large a sensor. Resolution and brightness both fall off at the edges. Heavy vignetting here. It's a decent lens for individual shots giving an artistic effect, but not a good choice for a sharp clear picture.

Looking at the F-keys at the bottom you can somewhat make out the most interesting characteristic of this lens, and it's the way bokeh is rendered. Blurring is somewhat radial about the center of the lens, and produces a unique effect with background lights.

Distortion is pretty bad, but you know what you're getting into with this lens. Again - One of my favorites, but you have to work within the character of the lens.

This Pentax is the cheapest of lenses here - not just in cost really (this was purchased new) but in design, feel, and optical quality. A Modern c-mount. The very tips of the corners have vignetted to almost nothing. Like the Wollensak above, there's distortion, vignetting and it gets worse at the edges but unlike the Wollensak, the center isn't nearly so spectacular, and of course we see a characteristic glow on object edges with this lens used wide open. Cheap lenses like this often exhibit a curving focal plane - i.e. as you move from the center of the lens to the edge, an object may have to move closer or further away from the plane of the lens and sensor to stay in focus.

Of course, the question here is "is this usable?" - again like the Wollensak, there's a lot of "character" i.e. imperfections you need to work with. It can give you a decent picture if you try. The redeeming aspect of it is it's weight, and ease of focusing. the focus moves smoothly and easily between the index finger and thumb.

So that's 4 lenses - 3 different types that are 25mm F1.4. There's no MTF curves or going into technical details here and I'm only looking at them wide open - so this article's usefulness will be rather specific, but I think it's interesting to see them here. As evidenced, everything is a tradeoff. Size, cost, build quality, ease of focus, sharpness, vignetting, chromatic aberrations - even the high end Leica here loses to all 3 in ease of manual focus, and the 35mm easily has it beat for vignetting. Every lens here has something it's better than the others at - even the optically worst lens of the bunch has redeeming qualities. Photographic gear is an exercise in compromise. 

I have a crapload (technical term for a lot) of lenses with overlap. But it doesn't mean I don't have a reason to use them. Sometimes I run with the Wollensak just for fun. Since I'm heavy into manual lenses, it's good to keep practicing focus. The results are different, and can be quite fun when done right - say, taking a photo of someone in front of a Christmas tree and seeing the swirly bokeh transform the background of image into a more atypical picture.


Everyone with an interchangeable lens camera should try a manual focus lens sometime. You may like it - and if you do, you suddenly open up decades of easily adaptable glass, often with wider apertures that you'd normally get as a result of cost - and with a focal reducer, that F-Stop can become even better, while supplying a wider angle of view.

Wollensak 25mm F1.5 - "Some dude on a bike..."
Sharp center makes the bike "pop". Less sharp edges don't matter here, and the distortion is somewhat masked by motion blur.