Tuesday, June 10, 2014

Backup your data

When you've got a collection of files - pictures, videos, documents etc, it's a pain to lose it. If it's personal - Pictures and videos of family for instance, then it's priceless.

I recently lost a couple drives in my main PC. Thankfully, a backup on another machine pulled through.

There's multiple solutions that can be engineered. A good solution will backup data on another remote server automatically.

Backup on the same machine is ill advised. In fact, let's not call this backup...

If it's the same drive, it's not very useful at all. I only do this if I'm editing a file and want the original as I edit. I certainly don't do this for storage.

If it's on another drive it's better, but not ideal. I've lost data on multiple drives at once. Sometimes it's not the drive that goes bad. A faulty harddrive controller was responsible for the first incident. Recently I lost data on multiple drives again, due to the operating system driver becoming corrupt. Once formatted and reloaded, new drives worked just fine.

It's nice to be able to recover data I removed intentionally. If I've deleted a RAW or video and want it. It's also nice to be able to recover data fully or initiate backup procedures at will.

let's consider Synctoy.

Synctoy is a Microsoft tool for syncing folders.
Get it here 

It's pretty easy to understand. Syncs can be done in either direction, or even both ways depending on date. Errors in copying files will not cause it to stop.

Since this works from folder to folder, you're not really limited to syncing this from a Windows workstation to windows. You can have your alternate run Mac, Linux, Solaris - it doesn't matter so long as you share the backup drive, and map it in the windows machine.

Open explorer, and click on tools->Map network drive.
The above gui shows a very simple setup which will mount the "d$" share from server as drive "Z:" locally.

Now, when I'm browsing what looks like the "Z:" drive on my PC, it's actually the share from the other machine -

Great - So now you have details on how to establish connection to the backup server (map the drive) and setup the sync and run manually (synctoy).
Next - automate it.

Command line is where it's at for automation. 

C:\Program Files\SyncToy 2.1>SyncToyCmd.exe -R "backup"
Preview of backup (E:\, H:\) in time 00:00:26:712.
SyncToy action was 'Echo'
Found 33 actions to perform.
Found 120,270 files that did not require action.
Analyzed 4,502.4 files per second.
Avoided copying 2,779,264,431,572 bytes in 120,270 files.
Saved approximately 08:02:08:00 by not copying any files.

SyncToy run of backup (E:\, H:\) completed at 6/10/2014 12:12:28 PM.
SyncToy action was 'Echo'.
SyncToy options were:
        Active for run all
        All files included
        No files excluded
        Do not check file contents
        Include read-only files
        Include hidden files
        Include system files
        Backup older files (send to Recycle Bin)
        All subfolders included
SyncToy run took 00:01:00:425.
Copied 7,452,599,872 bytes in 21 files in 00:01:00:425.
Bytes per second 123,335,434.1, files per second 0.3.
Avoided copying 2,779,264,431,572 bytes in 120,270 files that did not require ac
Saved approximately 06:15:34:193 by not copying all files.
Warning: 8 failures occured.
You can retry by selecting "Run" again or select "Preview" to see
the operations that remain to be performed.

The Sync operation completed successfully on folder pair 'backup' but some files
 were skipped. Please look at the logs for more details.

OK - next step, automate that command, and display the errors for you.
Lets make a task:

Open Microsoft Task Scheduler and create a new task:

Key here is to run if the user is logged in or not.

Next - set the trigger as a scheduled time:

I've set this to run daily, at 2:00AM.

Finally, set the command:

I've created backup.bat to run the command:
::run backup
"C:\Program Files\SyncToy 2.1\SyncToyCmd.exe" -R "backup"

Click OK and you'll be prompted for your password to save the task.

There ARE some ways to alert yourself of errors with the backup.

After calling the synctoycmd.exe command to sync, we can have the bat file run a command to alert us only if there's a problem.

How can we run a command, IF the log contains "error"?
C:\>findstr "Error" %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToy
Log.log && echo "RUNS IF ERROR FOUND"

%LOCALAPPDATA% is the Windows Environment variable for the current user's AppData\Local folder.

This will check the synctoy log and if there's text returned, it will run the echo command. If not, it wont run the echo.
So lets say we wanted this to copy the log to the desktop:

C:\>findstr "Error" %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToy
Log.log && copy %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToyLog.
log %USERPROFILE%\Desktop\synctoy_ERROR.txt

Now, the log will grow continuously, so after this, it's probably best to rename the log, so that the old errors wont trigger a later successful backup.

e.g. the BAT file may look like this:

::run backup
"C:\Program Files\SyncToy 2.1\SyncToyCmd.exe" -R "backup"

::create error log on desktop if there's errors
findstr 'Error' %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToyLog.log && copy %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToyLog.log %USERPROFILE%\Desktop\synctoy_ERROR.txt

::archive the last log
type %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToyLog.log >> %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToyLog.log.ARCHIVE.txt

::delete the log so old data wont trigger the file copy 
del %LOCALAPPDATA%\Microsoft\SyncToy\2.0\SyncToyLog.log

i.e Backup, create error report, archive the log.
Edit: I've made a few changes to use environment variables.
Just copy the above text to a file - say "backup.bat" and run it at a scheduled time.

There are more obvious ways to let you know a backup has failed, such as displaying the errors in the log on the windows background using samurize.
Read more here at lifehacker.

Sunday, April 20, 2014

Evaluating the cheap focal reducer with "real world" images: part 2

I've been a bit tardy in this review. Poor weather and a busy job would to that I guess. This time though I've gotten a decent variety of images for testing 3 fast lenses with the cheap focal reducer.

The Bower (Samyang) 24mm F1.4 (17mm F1.0), Vivitar (Samyang) 85mm F1.4 (61mm F1.0) and a special case - Spiratone 135mm F1.8 (97mm F1.3).

So, here we have a wide 35mm Equivalent, short-tele/long portrait ~120mm equivalent, and telefocal ~200mm equivalent. When we say equivalent, we're simply comparing focal length of these lenses with what we would need on a 35mm "full frame" sensor to get the same angle of view.

For Part 1, click HERE.

These shots are from the Bower 24mm F1.4 (EF). As far as fast wide lenses go for full frame cameras, this is wide, fast and cheap. So you probably wont expect good optical quality, however I think anyone would be surprised to see how well it does. This is also a good choice for astro-photography.
ISO 200 1/2500 F2.8
ISO 200 1/100 F1.4

ISO 200 1/320 F2.0

ISO 200 1/800 F2.4

ISO 200 1/30 F1.4

ISO 200 1/25 F2.8

ISO 250 1/125 F2.0

Next up we have the Vivitar 85mm F1.4 (K). The lens has been modified slightly. Being a Pentax K mount, it will just work on any EF adapter really, but the lever gets in the way of the mirror on my 5D, so I clipped it off with a dremel. This lens is just fantastic on Full Frame. It's not too long to make it unwieldy, and the smooth bokeh is breathtaking. 
ISO 200 1/640 F2.8

ISO 400 1/10 F1.8

ISO 640 1/160 F1.4

ISO 200 1/500 F1.4

ISO 200 1/800 F1.4

Finally, here's pictures from the titan of the bunch, the Spiratone 135mm F1.8 (YS mount with F adapter). This lens is impressive. It's not much longer than the 85mm, but it's 82mm filter and massive elements immediately tell your brain to put some strength in your arm when picking this up. Then you're surprised because you still should have used a bit more. It's heavy, solid, and will built. It's all metal, with straight, square lines. While prone to flare, it can deliver sharp detail, although some processing may be needed to bring out the best from this beast.
ISO 200 1/1600 F1.8

ISO 200 1/1600 F1.8

ISO 200 1/2500 F2.8

ISO 200 1/1250 F1.8

I'll have to get around to posting how my EP3 looks with these lenses.

Hope you enjoyed this as much as I do :) GAS (Gear Acquisition Syndrome) can be costly, but ofter buying used lenses can be quite cheap.

The focal reducer has gone beyond my expectations. The wider focal lengths are more usable, and the stop extra more light is fantastic. The center of the image shows a small boost in sharpness as well, and since the image is wider, it's also less prone to shake giving a couple advantages for sharper pictures. I haven't found any increase in flare, or odd bokeh.

The 24mm in particular is a favorite of mine, and has supplanted the Panasonic Leica 25mm F1.4 as the "standard" lens on my EP3. It's a combination of the 25mm Focal length, and F1.0 speed.

Happy Shooting!

Thursday, March 13, 2014

Evaluating the cheap focal reducer with "real world" images: part 1

Focal reducers really set something off with the mirrorless world. Not only for Micro Fourthirds, which is my "main" system, but also APS-C systems from Fuji and Nex.

Under a Focal reducer, the APS-C systems are "full frame" equivalent, giving the same image with the same lens for a photographer. Micro Fourthirds still gives a bit of a crop with 1.44x, but that's still pretty usable.

I figured I'll try some older lenses and see what I can get with this focal reducer adapter. Not only can it be used with old Full frame lenses, but there's a couple APS-C lenses out there as well, which would be very well utilized to deliver a wide image. I don't have many APS-C coverage lenses, because most would not have an aperture ring. I do have a samyang 8mm fisheye for APS-C. Looking forward to their 10mm lens.

The following Shots are from a Vivitar 17mm F3.5 (Nikon F). The lens is among the widest available from old film days, and delivers decent quality stopped down to F5.6. On the focal reducer, it's still not that fast, bearing F2.5, but it is wider than most MFT lenses, coming in at 12.2mm

ISO 2000 1/25s F5.6

ISO 320 1s F5.6

ISO 3200 1/25s F5.6

These shots are from the 8mm Samyang (Nikon F). Though it's a recent APS-C lens, it's completely manual, including the aperture ring. It should be noted that my copy never focused properly, and has been modified to achieve proper focus. Using this fisheye is very fun. While it is possible to "de-fish" the image to a rectilinear the fun and utility of a fisheye lies in the distorted projection. Ultra wide rectilinear lenses show heavy stretching towards the edge of an image.

ISO 400 1/20s F3.5

ISO 400 1/100 F5.6

ISO 400 1/80s F5.6

ISO 200 1/6s

ISO 200 1/2.5s F3.5

ISO 200 1/6s F5.6

These shots are from the Kiron 28mm F2 (Nikon F). Of all my old lenses, the quality here is probably the lowest. It's not a BAD lens. Few lenses are actually "BAD" This lens when stopped down is decent, but wide open is a little soft. Colors come out fairly well though. My pictures are probably looking repetitive here since my "testing" is really on my way home from my day job. I suck at street photography, so here's what we get. The third picture here looks extremely soft towards the bottom. All in all, I think the lens is quite usable. These pictures look nice and bright when in reality it was fairly dark outside. This is an effective 20mm F1.4, perhaps I should run a test of this vs the venerable 20mm F1.7.

ISO 400 1/80s F2

ISO 400 1/50s F2.8

ISO 500 1/20s

ISO 1250 1/40s F2

ISO 640 1/40s F2

These are the Canon 55mm F1.2 SSC (FD). This mount was converted to EF using the popular "EdMika" conversion kit. This is where we leave behind common thought and see what a $500 40mm F0.86 lens will look like. The focal reducer and 50 F1.2 combine to form a shockingly fast and capable lens. Shots here range from wide open (F1.2) to F2.

ISO 640 1/40s

ISO 200 1/40s F1.2

ISO 400 1/60s F1.2

ISO 200 1/60s

ISO 500 1/60s F1.2

ISO 400 1/60s F1.4

Up next I'll be trying lenses like the 85mm F1.4 and 24mm F1.4 Samyang lenses. I decided to break it up a little since the page loading gets heavy with too many pictures. 

It's interesting to note that even fast lenses on the focal reducer can be sharp and while it wont have as much contrast as stopping down a little, they can have enough contrast.

NB: to add ISO and shutter speed on these pictures comments. Blogger strips EXIF (why?!) 

Tuesday, February 18, 2014

Scripting a HEX edit for unrecognized RAW files

It happens every once in a while. Someone buys a new camera with awesome bells and whistles, snaps away some shots and then as they start their workflow, they hit a file format wall. Often the format of the file is exactly the same as older cameras of the line, with a change to the camera type field. RAW converters use that field to determine settings for color profile, white balance, interpolation type etc.

So when your new Canon T4i isn't "recognized", but the application knew what to set for T3i and T2i cameras which used the same sensor, and family of image processors, you really wonder why there isn't an automatic setting you can simply configure to treat the RAW file like that of it's predecessors. It's something that makes sense, and more so, why can't you just download a text or XML configuration for the camera and import it. I'm looking at you Adobe...

A perfect world where software makes logical sense for the user doesn't exist. Logical sense for a commercial product mean getting users to update and spend money so perhaps that's a reason why importing a profile doesn't exist.

So let's get down to understanding what we can do.
  1. A previous model of camera is properly understood
  2. The name is what many applications use to recognize the format and apply profiles
  3. The profile would be almost identical for a new model of camera
  4. File format difference is minimal - a couple bytes that make up the name normally

Logically Lets just change those bytes!

NB: there are command line tools that will do this in a simpler way. I just choose to script because it's more satisfying ;)

To change the bytes of the RAW binary file, we can convert to HEX, perform a substitution, and reconvert to binary.

To convert a file to HEX, there's "xxd". It's a unix command, but also available on windows, through installing GVIM.

Tools needed:
SED : https://github.com/bmatzelle/gow/wiki
XXD : http://www.vim.org/download.php

We will make this easy to use in 3 steps:
  • Install tools SED and VIM (xxd)
  • Create BAT file
  • Put BAT file in "send to" folder for easy right click  menu

E.g. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Looking a a small random file "subnip.au3"
C:\temp>type subnip.au3
C:\temp>"c:\Program Files\Vim\vim73\xxd.exe" subnip.au3
0000000: 4d73 6742 6f78 2830 2c30 2c42 6974 414e  MsgBox(0,0,BitAN
0000010: 4428 3234 382c 3130 3829 29              D(248,108))

Great, that worked. We can use this format to identify what we want changed. Use the column on the right to easily read the text to change, and then determine what hex code to change.

For example, lets say I wanted to change 248,108 to 249,110.
The 2 starts on the third character of that line:
With the hex code, every 2 characters is one ASCII character, so your match is here:
4428 3234 382c 3130 3829 29
So matching up what we want changed:
4428 3234 382c 3130 3829 29
We'd need to rplace the highlighted areas with the text we want.

The UNIX tool "sed" is perfect for this.
first, let's simplify our xxd output.

C:\temp>"c:\Program Files\Vim\vim74\xxd.exe" -ps subnip.au3

Perfect - now it's easier to use sed. Let's match what we want:
4428 3234 382c 3130 3829 29
From this, the string we want to change is
44283234382c3130382929 to

i.e. we've changed to HEX representation of the ascii codes we want to use.
NB: This is a byte for byte replacement. We're not adding bytes, or removing bytes.

Lets put this all together!
C:\temp>"c:\Program Files\Vim\vim74\xxd.exe" -ps subnip.au3 | sed "s/44283234382c3130382929/44283234392c3131302929/" | "c:\Program Files\Vim\vim74\xxd.exe" -ps -r


Huzzah! Big success! The same technique can be applied to a larger RAW file to fix our camera support.


Now onto RAW HEX editing.
I was able to get RAW files from here:

In this script, I will take Olympus EM10 RAW, and change the camera type to EM5. EM10 is the new model, which is almost exactly the same sensor and processing as the EM5.

Start by running xxd against RAWs and dumping them to text files. Open in VIM and compare. Look for where the Camera name is kept, and what to change.

From examining HEX dumps in VIM, we can determine the strings to replace.
In converting EM10 to EM5, we're converting
452d4d3130202020202020202020 to

Let's make a script to take the incoming filename as a parameter and simplify the process.


:: go to "%appdata%\Microsoft\Windows\SendTo" in file explorer
:: drop this bat file in there

set xxdbin="c:\Program Files (x86)\Vim\vim74\xxd.exe"
set srt=%1

set fin=%1.em10conv.orf
echo converting %srt% to %fin%

set iem10=452d4d3130202020202020202020
set iem5=452d4d3520202020202020202020

%xxdbin% -ps -c 4096 %srt% |sed "s/%iem10%/%iem5%/"|%xxdbin% -ps -c 4096 -r > %fin%


Perfect - lets make it easy!
Navigate to "%appdata%\Microsoft\Windows\SendTo" in file explorer, and save the BAT file here.

After those 3 steps - install GVIM, GOW and bat file in "send to" -
Now when you right click on a file, you can "send to" this bat file.
The output will appear in the same folder with same name as the original+"em10conv.orf"

Overall the process isn't that bad really. It's just a several little things to put together.

Monday, February 3, 2014

A little test of 4 different 25mm F1.4 lenses on Micro Four Thirds

So, I'm not fond of formally testing lenses myself (though I do like to read em). Comparing a couple lenses may be fairly quick (quick and dirty may not check everything ), but putting four lenses into an easily readable report takes up some time. So I'm not doing this with too much depth. This is just to assert to myself the feel of these different solutions, that share the same 25mm f1.4 specification, and look at more obvious differences in their results.

This is a very different kind of roundup. There's only one lens here I'd expect people to come across "in the wild". The next most common is a C-mount so yes, lets get weird...

The lenses here (on my Olympus EP3) are:
  1. Panasonic (Leica) DG Summilux 25mm F1.4 ASPH.
  2. Pentax TV lens 25mm F1.4 (1" coverage)
  3. Wollensak Cine Raptor 25mm F1.5
  4. Nikon 35mm F2.0 AI Nippon Kogaku
First off, some of these names just simply sound exotic. It brings to mind images of awesomeness that simple named lenses can't match. Obviously, fancy names equal good image quality...
Sarcasm aside, I like these names. It really helps set these aside from just being a mundane amalgam of specifications.

About the lenses:

Summilux is what Leica calls their current batch of lenses with an F1.4 aperture. Leica itself is a luxury brand in cameras. Leica's been at this for a while - their compact landscape camera dates back to 1913. They've got history, and a name that carries weight, so it makes sense that Panasonic would want to ride that name. The lens is Panasonic made but designed or approved by Leica. This is a current production lens for the high grade MFT market.

The most pedestrian of names is the Pentax TV lens. Quality is a bit better than the cheap 25mm c-mounts lenses on ebay that go for $25-$50, but it's not that great. It is quite comfortable to use with it's smooth ring though. This lens still exists, but is now branded Ricoh. This is a current low cost CCTV lens. Stepless aperture is good for video. While Pentax is also steeped in optical history starting with spectacle lenses, a mass production and cheaper lens like this probably inherits little of their mojo.

Wollensak was making camera lenses as far back as 1902,and shutters from 1899. They pushed high speed photography and video with their fastax brand, developing on a Bell Labs spinning prism implementation taking frame rates up to 18,000fps in the 1960s. For their film cameras, they have a line of cine lenses (c-mount)that were good value at the time. The company went under in 1972, so the lens here is at least as old as that.

Nikon has been around since 1917, but at the time was known as Nippon Kōgaku Kōgyō Kabushikigaisha (say that ten times fast...). It stayed that way for a long time, but in 1988, someone thought that it was too much of a mouthful, so the company was renamed to Nikon Corporation. "Nikon" dates back to 1946 and was formed from the words "Nippon" and "Ikon". Ikon was a Zeiss brand. In 1959, Nikon started pushing their Nikkor F line of SLR cameras. The 35mm F2.0 in use is an optical design introduced by Nikon in 1962. In 1975 Nikon updated the lens to stop down to F22, and since my lens doesn't have that option, the age is between 1962 and 1975 - possibly closer to 1975 since Nikon initially used chrome noses instead of all black on their lenses.

The Panasonic Leica produces a beautiful rendering. It's sharp to the edge, and the bokeh is smooth. Focus is quick. Manual focus is horrible since the focus ring is electronically connected to the glass and not mechanical. however, this gives it a relatively compact size for this level of quality and speed. It should be noted - this lens would be auto corrected in the camera for certain traits (CA/distortion etc), so the JPGs used would favor this more than other lenses here.
Ye olde Nikon 35mm F2.0 on focal reducer. Slight difference in focal length. Quality is decent, but it's definitely not as sharp. I can't say it's purely the focal reducer or the lens, the fault is probably shared between the two.

That said, it's sharp enough to satisfy most and make a good picture. Bokeh is nice, maybe the nicest of the bunch. it's focused a few millimeters in front of the Panasonic, so blurring is a little different, but as you can see it's quite smooth.

Also of note - the entire image shows little or no light falloff. Even on a full frame camera, this lens was well known to deliver low vignetting.
I loved using this Wollensak lens. Before MFT had a lot of lenses available, people were adapting any fast prime they could find to get a decent F-Stop. Classic manual focus 50mm lenses were common, and cheap, but they were also an equivalent 100mm on this format. The field of view more specific in usage compared to a "normal". C-mount lenses like these 25mm lenses were perfect fits for the system - small, light, and some actually delivered decent quality.

The Wollensak here shows a very sharp center quality - surprisingly sharper than the 35mm F2.0 on focal reducer. while it's not quite F1.4, the F1.5 is close - however it was never meant to cover this large a sensor. Resolution and brightness both fall off at the edges. Heavy vignetting here. It's a decent lens for individual shots giving an artistic effect, but not a good choice for a sharp clear picture.

Looking at the F-keys at the bottom you can somewhat make out the most interesting characteristic of this lens, and it's the way bokeh is rendered. Blurring is somewhat radial about the center of the lens, and produces a unique effect with background lights.

Distortion is pretty bad, but you know what you're getting into with this lens. Again - One of my favorites, but you have to work within the character of the lens.
This Pentax is the cheapest of lenses here - not just in cost really (this was purchased new) but in design, feel, and optical quality. A Modern c-mount. The very tips of the corners have vignetted to almost nothing. Like the Wollensak above, there's distortion, vignetting and it gets worse at the edges but unlike the Wollensak, the center isn't nearly so spectacular, and of course we see a characteristic glow on object edges with this lens used wide open. Cheap lenses like this often exhibit a curving focal plane - i.e. as you move from the center of the lens to the edge, an object may have to move closer or further away from the plane of the lens and sensor to stay in focus.

Of course, the question here is "is this usable?" - again like the Wollensak, there's a lot of "character" i.e. imperfections you need to work with. It can give you a decent picture if you try. The redeeming aspect of it is it's weight, and ease of focusing. the focus moves smoothly and easily between the index finger and thumb.

So that's 4 lenses - 3 different types that are 25mm F1.4. There's no MTF curves or going into technical details here and I'm only looking at them wide open - so this article's usefulness will be rather specific, but I think it's interesting to see them here. As evidenced, everything is a tradeoff. Size, cost, build quality, ease of focus, sharpness, vignetting, chromatic aberrations - even the high end Leica here loses to all 3 in ease of manual focus, and the 35mm easily has it beat for vignetting. Every lens here has something it's better than the others at - even the optically worst lens of the bunch has redeeming qualities. Photographic gear is an exercise in compromise. 

I have a crapload (technical term for a lot) of lenses with overlap. But it doesn't mean I don't have a reason to use them. Sometimes I run with the Wollensak just for fun. Since I'm heavy into manual lenses, it's good to keep practicing focus. The results are different, and can be quite fun when done right - say, taking a photo of someone in front of a Christmas tree and seeing the swirly bokeh transform the background of image into a more atypical picture.

Everyone with an interchangeable lens camera should try a manual focus lens sometime. You may like it - and if you do, you suddenly open up decades of easily adaptable glass, often with wider apertures that you'd normally get as a result of cost - and with a focal reducer, that F-Stop can become even better, while supplying a wider angle of view.

Wollensak 25mm F1.5 - "Some dude on a bike..."
Sharp center makes the bike "pop". Less sharp edges don't matter here, and the distortion is somewhat masked by motion blur.