Using Facial Recognition to ID Photo & Finally Success After Twenty Years!

This photo only says Koppel on the back. The family matches Mary Kurta-Kappel and the 5 children she left Belgium with in 1910. Likely taken in what is now Inzenhof, Burgenland, Austria

I've been searching of for photos of my Kappel great-grandparents, and their children, for a little over 20 years. I've been asking my family, and distant cousins online, if they had any such photos? For many years I've done frequent Google searches, and searched through the photos at Ancestry hoping to find photos of my great-grandparents and 10 of their 11 children. I was given a photo of my grandfather by an Aunt when I was a child. My grandparents divorced when my father was a child, and my grandmother moved with her children to Los Angeles. My grandfather remained in Chicago. The family knew very little about the Kappel side of the family and had no photos of them. My father didn't look like my grandmother's side of the family, and neither do I. I've been dying to see what the side of the family we took after looked like? The family was from what is today Inzenhof, Burgenland, Austria.

A 2nd cousin match of one of my aunt's contacted me to tell me he had some photos. That was last May when I was in Ireland and not checking my email. Fortunately I got in contact with him again in September. Coincidentally another relative got in contact with me at Ancestry around the same time. She also had family photos. After all this time two people came forward with photos!

My aunt's 2nd cousin match, who lives in Austria, has dozens of family photos the problem is many aren't marked. How does one identify people in photos, many taken over 100 years old? I thought about that for a while and remembered facial recognition software could match up the faces. The FamilySearch Discovery site Compare-A-Face's feature was the only once I was familiar with; so I began comparing the marked photos with the unmarked photos.

The FamilySearch Compare-A-Face feature gives you a percentage which represents your likeness to the person you're comparing with. The site is designed to allow you to compare yourself with your ancestors and relatives. I found that it's actually even more helpful when it comes to comparing the same people in different photos at different ages.

What I needed to do after beginning my comparison was to decide what percentage would indicate the a person in two different photos was the same person? I haven't found a relative that I matched that was more than 60% similar to me, my father who looked much like his father had a 70% likeness to him. I've also found that even unrelated people can get a 60% likeness, which would lead me to believe that any match in that range isn't really significant. It appears to me that a likeness over 70% indicates a relationship between those compared. A likeness over 90% is a strong indication the people in the two photos are the same person (although siblings can match in that range).

The marked photos I have are mainly of people over 50 years of age. The unmarked ones are of people under 40. Here is an example using a comparison of my great-grandmother Mary Kurta-Kappel's photos. It gives a 97% likeness which I would take as a confirmation they are of same woman. If both photos were taken at around the same ages I would expect a 99% likeness. Below we see Mary on the left under 30 years of age, and Mary, on the right, in her 70's. There are still enough similarities too make an accurate identification. I've found this isn't always the case. Some people change quite substantially with age and their comparisons fall below 90%. Also the quality of the photos makes a difference. Poor quality images may not produce accurate results.

I had wondered how high a percentage would be possible among family members who look very much alike? I was saying that 90% scores generally confirm two people are the same, but actually siblings can also match in the 90% range at FamilySearch. That is because this feature at FamilySearch isn't designed for the purpose of identifying the same person in different photos. I think it is very helpful for doing that in most cases because most people aren't carbon copies of other people. Siblings can be an exception, however. 

I decided to try the Amazon AWS site to see if it would help me differentiate people that look very much alike such as these siblings below. The FamilySearch Compare-A-Face feature says they are 92% similar and we might conclude they are the same person? No they are not. They are Kurta family sisters Mary and Rosalia.

The facial comparison feature at Amazon is called Rekognition. It's used by law enforcement agencies to identify criminals. Doing a comparison of the same photos with the same people above Rekognition isn't giving a percentage score in the 90's or 80's, for that matter, because the strike mark through the equals signs means they are unlikely to be the same person at all. A close look at their facial features confirms to me this is a correct prediction.

Amazon's facial Rekognition isn't very good when it comes to low quality images or people compared at different ages. I'm still using both sites for my comparisons. I feel very confident when both sites agree with nearly the same high percentages.

The comparisons are especially difficult when it comes to children. Apple suggests children under 13 avoid using facial recognition on their phones because children, especially siblings, can look too much alike. Apparently children don't develop an appearance with enough distinctiveness until puberty for facial recognition to be a secure lock for devices.

Here you can see how 2 siblings of different sexes are so similar looking they are both given high percentages of likeness by Amazon Rekognition. The photo of the family group is marked Koppel on the back with no first names. The mother in the photo matches Mary Kurta-Kappel at a high percentage so I would say this is Mary and her children. Mary and 5 of her children sailed from a port in Belgium to New York in 1910. These children seem to match those children age wise (maybe a goodbye photo?). The eldest boy was Stephen. I would guess the boy in both photos is likely Stephen at different ages.

We also see faces that aren't as similar with the = sign struck through. 

Another match between this photo and an unmarked photo is the one below. It appears the eldest, daughter in the family group photo matches another photo of a woman with her husband and baby. I wouldn't have caught this without facial recognition. Once I saw the high percentage I looked at the two photos very carefully and the look of the eyes, shape the face, and mouth do look very much alike. There is a possibility of error when it comes to using computer generated facial comparisons. You have to examine the photos yourself and decide whether the predictions are correct? As I have read humans are still better than computers at picking up subtle differences in faces. 

One feature which is helpful that FamilySearch Compare-a-Face doesn't have, and Amazon's Rekognition does, is you can find someone using a single photo of them uploaded on the left and a group photo on the right. An example of this is the comparison below. I used a photo of Johann Koeppel at his wedding and a group photo of his sister Juliana's wedding. This feature successfully picked Johann out of the crowd, and eliminated the rest of the group. 

I have a few people in photos I haven't be able to identify. Here is an example of how I'm trying to identify these people. I have a photo of 3 people that is unmarked, below, or it is stuck down with glue so you can't see the back? According to what is written on this photo page 3 siblings may appear in photos on this page. They are named as Julia, Herman, and Maria/Mary (Mary is my ancestor). A photo at the top of the page appears to be of 3 siblings. We know Herman is the man in the photo but the two women are trickier. According to facial recognition their faces are similar to one another with high percentages of similarity. My observation is the woman on the right is Maria/Mary. So is the woman on the left Julia? I have no marked photos of her so I don't know? I would like more verification before I make a judgement. Since the two women look alike I would guess they are very likely Mary and her sister Julia? 

When comparing the unknown person in the photo above with another unknown in another photo there are similarities. The wedding photo, below right, was obviously taken many years before the photo above so there is unlikely to be an exact likeness. According to FamilySearch's Compare-A-Face they are 85% similar. I've verified with other family photos that can be the case as people age they often compare with younger photos in the 80% range. However, Amazon's Rekognition disagrees, and does not identify them as possibly the same person. Like I said before Amazon's feature doesn't do as well with poor quality images or people at different ages. Unlike the FamilySearch feature this product is designed to identify the same person at around the same age. 

Care always has to be taken because even cousins look alike. I have enough identified pictures of Mary/Maria Kurta-Kappel to identify the photo below with a high degree of confidence that this is Mary Kurta-Kappel with her husband Frank Kappel. It better be  Frank or Mary was cheating lol. 

I can further identify Frank by comparing him his cousin who is in marked photos. I thought he looked somewhat like his cousin; so I did the comparison. Compare-A-Face agrees with me. When I compared at Rekognition it did its job by declaring they were not the same person, and clearly they are not. However I now know cousins can have similarities in the 80% range. 

Doing so many comparisons has helped me understand how these features work, and how to best use them for identifying unknown people in photographs, and even people who were wrongly identified in photos. 

Amazon's AWS Rekognition requires that you provide credit card information to use it. This feature is free to use. Other features on this site cost money to use so they want a credit card on file because of that. I think they want to know who is using this also? They'll charge your credit card a $1 fee to verify it is good. They are supposed to refund the dollar but this is so helpful I don't care if they do. It's worth $1. I had a hard time logging in the first time. You have to click the root login then enter your email address which brings up the box for your password. 

The only requirement for using the FamilySearch Compare-A-Face feature is a free account with them. The only draw back with using this site to compare multiple photos is when you want to change the primary person for comparison you have keep clicking menu in the top right hand corner to go back to the upload page (it's helpful to save images when asked although it doesn't always work?). You can compare with the primary person's photo, on the right, continuously by clicking the plus sign in the left corner to upload a photo or compare with saved photos shown across the top of the page. . 

I am able to compare some of the photos using my own eyesight. People at different ages, and people who have gained or lost weight, can be tricky for me to compare using my vision alone. The facial recognition comparisons have been very helpful for me as I sort through dozens of photos. 

Never give up if you are also looking for photos of ancestors. You never know who might have them, or where they might be? Remember in the 20th Century our ancestors attended events where they may have been photographed, such as family weddings? Also my great-grandmother sent photos of herself and her family back to her sister who still lived in Austria. Keep looking and asking if you're looking for family photos. 

Below are some photos I can't identify. They were taken in Allentown, Pennsylvania. They are likely Kurta family relatives. One may be Julia Deutsch nee Kurta?