Scripting a HEX edit for unrecognized RAW files

It happens every once in a while. Someone buys a new camera with awesome bells and whistles, snaps away some shots and then as they start their workflow, they hit a file format wall. Often the format of the file is exactly the same as older cameras of the line, with a change to the camera type field. RAW converters use that field to determine settings for color profile, white balance, interpolation type etc.

So when your new Canon T4i isn't "recognized", but the application knew what to set for T3i and T2i cameras which used the same sensor, and family of image processors, you really wonder why there isn't an automatic setting you can simply configure to treat the RAW file like that of it's predecessors. It's something that makes sense, and more so, why can't you just download a text or XML configuration for the camera and import it. I'm looking at you Adobe...

A perfect world where software makes logical sense for the user doesn't exist. Logical sense for a commercial product mean getting users to update and spend money so perhaps that's a reason why importing a profile doesn't exist.

So let's get down to understanding what we can do.

1. A previous model of camera is properly understood
2. The name is what many applications use to recognize the format and apply profiles
3. The profile would be almost identical for a new model of camera
4. File format difference is minimal - a couple bytes that make up the name normally

Logically Lets just change those bytes!

NB: there are command line tools that will do this in a simpler way. I just choose to script because it's more satisfying ;)

To change the bytes of the RAW binary file, we can convert to HEX, perform a substitution, and reconvert to binary.

To convert a file to HEX, there's "xxd". It's a unix command, but also available on windows, through installing GVIM.

Tools needed:
SED : https://github.com/bmatzelle/gow/wiki
XXD : http://www.vim.org/download.php

We will make this easy to use in 3 steps:

• Install tools SED and VIM (xxd)
• Create BAT file
• Put BAT file in "send to" folder for easy right click  menu

E.g. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Looking a a small random file "subnip.au3"
C:\temp>type subnip.au3
MsgBox(0,0,BitAND(248,108))
C:\temp>"c:\Program Files\Vim\vim73\xxd.exe" subnip.au3
0000000: 4d73 6742 6f78 2830 2c30 2c42 6974 414e  MsgBox(0,0,BitAN
0000010: 4428 3234 382c 3130 3829 29              D(248,108))

Great, that worked. We can use this format to identify what we want changed. Use the column on the right to easily read the text to change, and then determine what hex code to change.

For example, lets say I wanted to change 248,108 to 249,110.
The 2 starts on the third character of that line:
D(248,108))
With the hex code, every 2 characters is one ASCII character, so your match is here:
4428 3234 382c 3130 3829 29
So matching up what we want changed:
D(248,108)) 
4428 3234 382c 3130 3829 29
We'd need to rplace the highlighted areas with the text we want.

The UNIX tool "sed" is perfect for this.
first, let's simplify our xxd output.

C:\temp>"c:\Program Files\Vim\vim74\xxd.exe" -ps subnip.au3
4d7367426f7828302c302c426974414e44283234382c3130382929

Perfect - now it's easier to use sed. Let's match what we want:
4428 3234 382c 3130 3829 29
4d7367426f7828302c302c426974414e44283234382c3130382929
From this, the string we want to change is
44283234382c3130382929 to
44283234392c3131302929 
i.e. we've changed to HEX representation of the ascii codes we want to use.
NB: This is a byte for byte replacement. We're not adding bytes, or removing bytes.


Lets put this all together!
C:\temp>"c:\Program Files\Vim\vim74\xxd.exe" -ps subnip.au3 | sed "s/44283234382c3130382929/44283234392c3131302929/" | "c:\Program Files\Vim\vim74\xxd.exe" -ps -r

MsgBox(0,0,BitAND(249,110))

Huzzah! Big success! The same technique can be applied to a larger RAW file to fix our camera support.


 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=

Now onto RAW HEX editing.
I was able to get RAW files from here:
http://www.imaging-resource.com/PRODS/olympus-e-m10/olympus-e-m10A7.HTM

In this script, I will take Olympus EM10 RAW, and change the camera type to EM5. EM10 is the new model, which is almost exactly the same sensor and processing as the EM5.

Start by running xxd against RAWs and dumping them to text files. Open in VIM and compare. Look for where the Camera name is kept, and what to change.

From examining HEX dumps in VIM, we can determine the strings to replace.
In converting EM10 to EM5, we're converting
452d4d3130202020202020202020 to
452d4d3520202020202020202020

Let's make a script to take the incoming filename as a parameter and simplify the process.

::+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

:: go to "%appdata%\Microsoft\Windows\SendTo" in file explorer
:: drop this bat file in there

set xxdbin="c:\Program Files (x86)\Vim\vim74\xxd.exe"
set srt=%1
set fin=%1.em10conv.orf
echo converting %srt% to %fin%

set iem10=452d4d3130202020202020202020
set iem5=452d4d3520202020202020202020

%xxdbin% -ps -c 4096 %srt% |sed "s/%iem10%/%iem5%/"|%xxdbin% -ps -c 4096 -r > %fin%

::+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Perfect - lets make it easy!
Navigate to "%appdata%\Microsoft\Windows\SendTo" in file explorer, and save the BAT file here.

After those 3 steps - install GVIM, GOW and bat file in "send to" -
Now when you right click on a file, you can "send to" this bat file.
The output will appear in the same folder with same name as the original+"em10conv.orf"

Overall the process isn't that bad really. It's just a several little things to put together.