Welcome to the Birder's Diary Forum for Support And General Questions
Use the Support forum for all questions or issues.
Use the Wish List forum to leave your ideas for improving Birder’s Diary.
Use the Community Sharing forum for sharing Photos, Trips, Stories, etc.
Setup your Forum photo and profile here.
I just downloaded and installed the new IOC 13.1 2023 list. The installer ssemed to behave normally, and went through the usual reconciliation step to deal with splits. However, when I got the reconcilation report, I had gained a few species, as expected, but I had lost 24, which seemed high. Looking at the list, at least some of these species (I did not check them all) have disappeared altogether from the new IOC list, and are not just renamed through a lump. To take a simple example, when I try to show my sightings of Eastern Meadowlark, using the new 13.1 list, no sightings appear at all. When I look at a day with Eastern Meadowlark sightings using the old IOC 11.2 list, I can see Eastern Meadowlarks. When I show the list for the same day using the new IOC 13.1 list, there is no Meadowlark of any description at all, just a shorter list. In other words, it does not seem to have been assigned a different name - there is just nothing corresponding to those sightings.
Here is a dropbox link to the Reconciliation Report:
https://www.dropbox.com/s/azbtsmzkgaynpso/ReconcileReport.html?dl=0
More details: IOC 13.1 does include the split of Eastern Meadowlark (Sturnella magna) into Chihuahuan (S. lilianae) and Eastern (still S. magna), and both taxonomic list and Rosetta Stone genus information properly include all three Sturnellas. However, this split was not included in the list of species to reconcile in Birder's Diary. All the sightings that WERE Eastern Meadowlark in IOC 11.2, whether they would now be Chihuahuan or Eastern no longer appear in lists when I use the IOC 13.1 taxonomy. Old sightings no longer correspond to a Thing in IOC 13.1. My hypothesis is that the other mysterious absences are all cases where there was some sort of split that did not get dealt with in the Birder's Diary reconciliation step.
More details: IOC 13.1 does include the split of Eastern Meadowlark (Sturnella magna) into Chihuahuan (S. lilianae) and Eastern (still S. magna), and both taxonomic list and Rosetta Stone genus information properly include all three Sturnellas. However, this split was not included in the list of species to reconcile in Birder's Diary. All the sightings that WERE Eastern Meadowlark in IOC 11.2, whether they would now be Chihuahuan or Eastern no longer appear in lists when I use the IOC 13.1 taxonomy. Old sightings no longer correspond to a Thing in IOC 13.1. My hypothesis is that the other mysterious absences are all cases where there was some sort of split that did not get dealt with in the Birder's Diary reconciliation step.
You are correct. And my response below corrects all of this. Let me know which option you would like me to use.
I can get onto your computer and run the code (it takes 10s) to correct this linkage. Or one of the other options.
Hi David,
Thanks for posting and for posting the Reconcile report. Very helpful.
There were a lot of Lumps in IOC 13.1. They went from split-happy to lump-happy this time around.
So, that report is what I would expect it to look like if you had not reconciled your Eastern Meadowlark sightings. But I suspect that if you pull up the Reconcile window (Features | Advanced | Reconcile Sightings) that you have zero Things to reconcile when you specify IOC 11.2 and IOC 13.1 - correct?
Then I realized that you have the Everything license, and so you first upgraded to Clements 2022 last fall and then just recently upgraded to IOC 13.1. These two taxonomic lists are not compatible in a handful (about 221 out of 31,000+) of Things. The Eastern/Chihuahuan Meadowlark is one of those. It is just a mind-numbing task to attempt to align everything between the two. The recommendation has always been to stick with one, and only one, for your sightings counts, etc. You can use the other to review differences, etc. but they won't always agree at a base level.
I have to weigh how many folks this might affect (99% of everyone sticks with 1 tax list), the effort to align most of the mismatches (2-3 weeks of mind-numbing comparison), potential other higher-severity errors that this alignment might cause with features such as Reconcile, and the value that doing such might add.
The equation comes out to allowing for 221 things that don't line up. What you have found.
I actually have code that will correct 99% of those 221 misaligned things. And since you have already done the Reconcile (first with Clements and now with IOC), the risk of side-effects is low.
I have been doing some testing now since your original post on this, and believe that the risks are lower than I thought (in fact, I couldn't find any issues after thorough testing), and that I should have done this alignment - as I have done in the past.
So now my plan is to realign the IOC 13.1 with Clements 2022 on these 221 things and re-release it. This will only affect <1% of BD users that actually import and somewhat use both taxonomic lists. So anyone that doesn't use both Clements 2022 and IOC 2023, there is no need to reimport.
For you, I can get onto your computer (or you can upload your DB to me) and I can run the code that will properly align these things. I tested the Reconcile report and test sightings and everything appears correct once I run this code, before or after the import/reconcile of IOC 13.1. Let me know what you want me to do with respect to your DB. You also could wait until I release this change, give me about 3 hours, and revert your DB back to pre-IOC2023 and reimport with the new IOC2023, and reconcile. Your choice. Let me know.
QED
Update: I should have been clearer about the code I already had that located mismatches between Clements and IOC. This code has existed for years now. However, due to changes in taxonomic lists, import process, reconcile process and the reconcile report... I wasn't sure that this code wouldn't break multiple things in multiple places. I reasoned that it was too much of a chance to use it for this release.
The problem is that I didn't try it as I didn't have a good idea on how to test it end-to-end in multiple places and with multiple different scenarios. But this post prompted me this morning to spent some time looking at this again, and it turns out the test to find out what it would break and how to fix it turned out not to be too hard. And the result is, it doesn't break anything. So, it is the right thing to do.
I have run this code against IOC 13.1 2023 and rebuilt the installer with the new files. Since David created a backup before importing this, he will restore to that backup and download/import/reconcile with the newly updated list.
If you do not use Clements and IOC interchangeably in Birder's Diary, then this is of no consequence, and there is no issue proceeding with the first release of IOC 13.1 2023.
Else if you do use both interchangeably and you have a backup from before the import of IOC 13.1, restore to that backup, download, import and reconcile again. If you do not have a backup, then remove IOC 13.1 from your system using the Taxonomy Viewer.
Now, download, import and reconcile IOC 13.1 again.
The new version of IOC 13.1 2023 is now available.
After some effort (details below for Jeff, and probably not of interest to anyone else), I got a much more complete conversion from IOC 11.2 to IOC 13.1. However, after careful checking, there are two remaining problems with my sightings that I do not know how to fix.
First, there was a complicated four-way split of Mouse-colored Tyrranulet in IOC13.1, and I had seen three of the four products of that split. However, only the Tumbesian Tyrranulet came across into IOC13.1. My sightings for the Northern Mouse-colored Tyrranulet and the Maranon Tyrranulet are not accessible in IOC 13.1.
Second, IOC 13.1 does not recognize any of my sightings of Forest Robin from IOC 11.2. This may be a different sort of error, as the genus level description for Stiphornis in IOC 13.1 brought up by Rosetta Stone has only a single species S. erythrothorax, the Forest Robin, but the text describes a three-way split. So, there is no actual separate description or Latin binomials of the products of the split: Olive-backed, Orange-breasted and Yellow-breasted Forest Robins.
This brings up two questions: First is there a way to bring these lost observations into IOC13.1? One idea that occurs to me again is to rewind to the pre-IOC13.1 backup, but then convert from Clements 2022 into IOC13.1, but I wanted to ask your opinion before trying that.
Second, I am worried that other sightings may have similarly been ‘lost’ in previous updates without me realizing it. Is there a way to detect sightings that have no Thing associated with them?
Here are the detailed obstacles that I had along the way to the above result. , I restored to the pre-update IOC13.1 Birder's diary database, updated to build 26, then downloaded the IOC13.1 taxonomy again. I ran the Birds-IOC-2023.exe file, which generated this error, which I believe I have seen with other taxonomy updates:
I clicked that away, and then Birder’s Diary opened. The update was shown in the Import New taxonomic lists window, and I clicked on AutoImport. Then two weird things: first, there were only two items to reconcile this time Eurasian Collared Dove and Wooly-necked Stork, so the majority of things I had to reconcile from my first time through were not there. Second, Birder's Diary crashed when I submitted the Wooly-necked Stork reconciliation
When I restarted Birder's Diary, I deleted the partially installed 13.1, closed Birder’s Diary, and rebooted for good measure. Then I reran the Birds-IOC-2023.exe, and when I reopened Birder’s Diary, the new taxonomy dialogue was there, and after running it, there were now NO reconciliations to act on. It seems like the old reconciliations were not lost when I restored from a backup made BEFORE I started on IOC 13.1. This seems pretty odd, but maybe not really a problem, unless is somehow leads to the remaining problems with my sightings.
Forgot to add the Reconcile Report from the latest:
https://www.dropbox.com/s/azbtsmzkgaynpso/ReconcileReport.html?dl=0
Hi David,
Thanks for all the details. I haven't read through it all yet, and just taking this from the top down.
This is going to be long and complicated, so please take your time, and let me know what questions you have.
First, there was a complicated four-way split of Mouse-colored Tyrranulet in IOC13.1, and I had seen three of the four products of that split. However, only the Tumbesian Tyrranulet came across into IOC13.1. My sightings for the Northern Mouse-colored Tyrranulet and the Maranon Tyrranulet are not accessible in IOC 13.1.
From what I can see, it is a 2-way split of Mouse-colored Tyrranulet for both IOC 2023 and Clements 2022. The reason you appear to be missing those sightings is because my IOC-2-Clements matching algorithm isn't robust enough to catch it. Here comes the mind-numbing, endless-hours part I was talking about.
Clements split from N. murina to N. incomta and N. murina.
IOC split from N. murina to N. incomus and N. murinus.
So, the matching program doesn't catch that difference. There are probably at least a dozen of these edge cases; and then a few more that fall outside of the possibility of the matching program.
Second, IOC 13.1 does not recognize any of my sightings of Forest Robin from IOC 11.2. This may be a different sort of error, as the genus level description for Stiphornis in IOC 13.1 brought up by Rosetta Stone has only a single species S. erythrothorax, the Forest Robin, but the text describes a three-way split. So, there is no actual separate description or Latin binomials of the products of the split: Olive-backed, Orange-breasted and Yellow-breasted Forest Robins.
IOC 2023 did not split this species, but Clements 2022 did split this species three ways into the above mentioned birds.
So, you cannot correctly assign IOC's S. erythrothorax sp to any of the 3 Clements spuhs.
So what I have done here is:
- Clements sp S. pyrrholaemus is assigned to IOC ssp S. e. pyrrholaemus - correctly.
- Under the two polytypic splits in Clements, each of their ssp are assigned to IOC's relevant S. erythrothorax ssp.
All good at the ssp level across both tax lists.
And here is where this problem lies... the matching program matched IOC's S. erythrothorax sp to Clements sp of the same classification. And, I think if you run/view your Reconcile report for Clements 2022 vs Clements 2021, you will probably see that you assigned any S. erythrothorax to S. mabirae or S. pyrrholaemus, but not to S. erythrothorax. Let me know. This would explain why you can't see any of your older sightings, as nothing in IOC 2023 is attached to those two spuhs in Clements 2022. Clements S. erythrothorax should not be assigned to IOC's S. erythrothorax. So we just found another edge case where the matching program did the wrong thing.
And now we are starting to see why no other programs ever attempt such an alignment.
This brings up two questions: First is there a way to bring these lost observations into IOC13.1? One idea that occurs to me again is to rewind to the pre-IOC13.1 backup, but then convert from Clements 2022 into IOC13.1, but I wanted to ask your opinion before trying that.
I am glad you did ask. But I don't see how that would help anything. And might only serve to muddy the waters with further issues.
With my description of the issue above, I think the proper course of action would be the following in the case of the Forest Robin.
- De-couple Clements S. erythrothorax from IOC's S. erythrothorax.
- Add sightings for IOC's Forest Robin at the spuh level. Clements won't see those.
- OR - - Move your sightings to the subspecies level where the assignments are correct and matching between the two lists.
Second, I am worried that other sightings may have similarly been ‘lost’ in previous updates without me realizing it. Is there a way to detect sightings that have no Thing associated with them?
There is no such thing as "Sightings that have no Thing associated with them", as far as I know. I don't see how that could happen, and we don't have any evidence of that here as far as I can see.
I ran the Birds-IOC-2023.exe file, which generated this error, which I believe I have seen with other taxonomy updates: ... file already in use ...
Running Birds-IOC-2023.exe doesn't touch the database.mdb file. So, I don't know how you could have gotten this error by running that. Very strange. The solution is to close BD, assuming it is running, and then run the installer again if that occurs again.
When I restarted Birder's Diary, I deleted the partially installed 13.1, closed Birder’s Diary, and rebooted for good measure. Then I reran the Birds-IOC-2023.exe, and when I reopened Birder’s Diary, the new taxonomy dialogue was there, and after running it, there were now NO reconciliations to act on. It seems like the old reconciliations were not lost when I restored from a backup made BEFORE I started on IOC 13.1. This seems pretty odd, but maybe not really a problem, unless is somehow leads to the remaining problems with my sightings.
The Not Responding error is probably a Windows issue. Stopping and restarting was the correct thing to do.
Not sure how the Reconcile info was retained if you restored to a backup from before the 13.1 import/reconcile actions.
So, in short, these last 2 things are strange anomalies, and let's ignore them unless they happen again. They will get in the way of the bigger picture here.
Let's focus on making the linkage between Clements and IOC as good as possible.
My thoughts on this in no particular order:
- Review your Clements Reconcile report. Hopefully that data is not compromised in any way. Please share that with us if you can. If it is good, it should give us the complete picture of how we got where we are.
- Review your Forest Robin and Mouse-colored Tyrannulet sightings using Clements 2022. When in the Sightings View/Edit window, use the slider to view all sightings under the Stiphrornis genus. Right click on the various sightings for the different Things and choose Thing Cross-reference from the popup menu. This will allow you to see what Things in IOC each is attached to.
- Think about the case of the Forest Robin, not split in IOC but 3-way split in Clements. Easy to assign the subsps but the species don't line up. I could create a tool that allows the user to correct Thing-2-Thing assignments between tax lists, but worry about creating a mess. Also, just how would you handle this specific situation of the Forest Robin?
- I don't think the Reconcile logic and code may break down under this situation - Clements-and-IOC - but I am not sure and don't have any evidence yet that it has. But for sure, I didn't design it with this possibility in mind.
That's enough for now. Let me know your thoughts.
I don't understand some basic things here, so before I do anything more I want to ask some more questions:
I did not keep older Reconcile sightings reports, so I can't go back and look at my Clements 2021 to 2022 for example. Your post implies that I can rerun the reconciliation, but I want to check whether that this won't cause more problems before doing it.
What I mean by a sighting without a Thing, is like the earlier case of the disappearing Meadowlark, of the current Forest Robin. I call for sightings in a particular taxonomy, and some sightings are simply not accessible. What I am worried about is the compounding effect of these changes. If converting from taxonomy to taxonomy only acts on the sightings referenced by that current taxonomy the next conversion will also miss them. So, if I did not correct Forest Robins now, they will never appear again, and they are effectively lost to me, at least within IOC. That is what I want to detect from previous updates, whatever you want to call it.
Hi David,
I did not keep older Reconcile sightings reports, so I can't go back and look at my Clements 2021 to 2022 for example. Your post implies that I can rerun the reconciliation, but I want to check whether that this won't cause more problems before doing it.
Just open the Reconcile window, fill in old and new Clements list, and then click on the Report button. That is all that is required. The report is built from an internal log that records every action you made during the reconcile.
The concern I have is that I didn't write the Reconcile functionality (including report) with the idea that someone would reconcile the same sightings across multiple taxonomies that disagree with each other to some extent.
What I mean by a sighting without a Thing, is like the earlier case of the disappearing Meadowlark, of the current Forest Robin. I call for sightings in a particular taxonomy, and some sightings are simply not accessible.
Sightings are *never* destroyed by BD except by explicit user action. Reconciled sightings are never deleted.
During the Reconcile, when the old taxonomy has a Thing which is *not* in the new taxonomy, the user is asked to assign those sightings to something in the newer taxonomy. Copies of those sightings are made and assigned to the user choices. An internal log is kept of all such assignments that the user makes.
So, in this example, Clements 2021 Mouse-colored Tyrannulet was split into Northern/Southern species in Clements 2022. The Thing assigned to 2021 Mouse-colored Tyrannulet was not used in Clements 2022; two new Things were created for Northern/Southern Mouse-colored Tyrannulet in 2022. So, Reconcile asks you to assign all existing sightings for Mouse-colored Tyrannulet in 2021 to one of the two new Things. And you do so and move on. And these assignments are logged.
In comes IOC 2023. It does not contain the old Mouse-colored Tyrannulet Thing nor the two new Northern/Southern Mouse-colored Tyrannulet Things; because they were not matched across the tax lists by the IOC-2-Clements matching code due to different sci names. So now, you have assigned your sightings to two Things that are in Clements 2022, but are not found in IOC 2023. So, when using IOC 2023, you don't see those sightings because nothing in that tax list classifies those two Things. Just as you wouldn't see sightings for a Monarch Butterfly, because that Thing is defined in the Butterflies taxonomy, and not in the IOC 2023 taxonomy. Important: the sightings are still there; they haven't gone anywhere; just inaccessible by IOC 2023; because IOC 2023 does not classify those Things.
What I am worried about is the compounding effect of these changes. If converting from taxonomy to taxonomy only acts on the sightings referenced by that current taxonomy the next conversion will also miss them. So, if I did not correct Forest Robins now, they will never appear again, and they are effectively lost to me, at least within IOC. That is what I want to detect from previous updates, whatever you want to call it.
Yes. That is going to happen with 200/10,000 (2%) of Things (only concerned about that 2% you have sightings for) when you mingle Clements and IOC. One taxonomic list has to be the Master, and the other for reference.
So, lots of details here. But I know you are the kind of guy that likes the details.
What I am thinking:
- The Reconcile Report already alerts you to missing sightings in this case.
- A tool that allows you to see the mismatches (potential issues) between any two taxonomic lists.
- A tool that allows a user to re-assign Things in one Tax list to match another. Concerned here because this can get users into unexpected issues if not thought through clearly before making any changes.
Let me know your thoughts.
@jeff First of all, I am well aware that the original sighting information is not being lost from the Birders' Diary database, so I don't need reassurance there.
I want to bring up two issues here. First is how to prevent these ghost sightings from escaping the first reconciliation where there is a name discrepancy. The second is how to detect older instances where sightings got lost in previous reconciliations. What I propose is making the user aware that there is a problem, which the user could then take actions to correct, for example, by reentering the usually relatively few sightings that this would affect. I also have a blue-sky idea for preventing these problems in future, but which might not be implementable from the current architecture of the database. but I will put that in a separate post.
As to the first problem of avoiding the initial loss of a sighting to a taxonomy. The Reconcile report as given now does not effectively do this. It gives list of things lost, and things gained, but not the correspondences betweeen them. A very useful addition would be to either give all transitions from one Thing to another, or just to flag the instances where one Thing in the old taxonomy has no corresponding Thing in the new taxonomy. Otherwise figuring this out is a very time-consuming task, and it is easy to misinterpret. For example, in my latest reconcilation Mouse-colored Tyranulet disappears, and White-fronted Tyrranulet appears, prompting one to think that Mouse-colored became White-fronted, but this of course was not true. They were two different splits, one that caused a succesful rename, and the other the effective loss of records.
Second, it would be great to have a tool to run that would look at all the sightings in the database and flag all those that do not have a Thing that corresponds to them in the taxonomy the user enters. Call it the Sighting-Thing mismatch tool.
In both of these cases, this would not involve you figuring out how to fix each and every problem in silico- it would just alert the user that there is a problem, and documentation could tell the user some generic solutions - like reentering the observations into the taxonomy in which it is missing.
Then there is the unfortunate case of the Meadowlarks where I have literally hundreds of sightings that were lost in my first IOC 13.1 update. This is far too many to expect to reenter. The more complex version of a resolution mechanism is that at the end of reconciliation, or upon running the sighting -Thing mismatch tool, the user would be presented with a list of sightings with their original names (or last known Things that matched them), and prompt the user to enter the appropriate name in the taxonomy they are missing from. Again, the burden woud be on the user to figure out which Forest Robin, matches which Forest Robin split.
I am going to continue to add more to this story/problem/puzzle, because I would like to improve this, and one of the ways is to get input/ideas for others.
Are you starting to feel the brain numbness yet? 😉
So, first some numbers.
- Clements 2022 has 31,294 Things (sp and subsp) defined. 10,906 are species; 20,388 are subsp.
- IOC 2023 has 30,970 Things (sp and subsp) defined. 11,140 are species; 19,830 are subsp.
- The 221 number I stated earlier needs clarification. The true definition of that number is the number of Things in each of the two taxonomies that share the same Sci Name but do NOT point to the same internal Thing in BD.
- This happens because of the algorithms that initially build the taxonomies don't take into account trying to align with other taxonomies.
- Now this tool that calculates these deltas and attempts to correct them is about 97% successful. But there are edge cases that don't fit simply auto-assigning IOC Things to Clements Things just because the Sci Names match. Sometimes IOC will switch the SciNames of Things, compared to Clements. For example, sometimes Clements will classify two Things as A. b. and A. c. while IOC will classify the same two things, respectively, as A. c. and A. b. But it is clear based on Range, that Clements' A. b. is IOC's A. c., and so forth. So assigning mismatched Things strictly on SciName is not always correct. And that is just one example.
- Other edge cases are like the Mouse-colored Tyrannulet, where both split but did not agree on the Sci Name for each in the split.
- Another is the case of the Forest Robin; 3-way split in Clements, not split in IOC. Clearly IOC's S. erythrothorax should not point to Clements' S. erythrothorax.
Now, let's add another tool that I created years ago in an attempt to send me to the insane asylum ahead of schedule. This tool identifies ALL THINGS classified in IOC that are not classified in Clements. That number is 789 as of right now. See screen shot here.
This was originally created for use with ABA/AOS which were subsets of Clements, and I needed to make sure all things in ABA/AOS were classified in Clements. When you run this tool against those 2 taxonomic lists, there are zero mismatches.
But the tool can easily be used for the purpose of identifying ALL mismatches between IOC and Clements. Trying to use this tool to correctly realign mismatches between the two taxonomic lists is mind-numbing!!! Easy to use to correct found edge cases such as the Mouse-colored Tyrannulet and Forest Robin. But, trying to process all 789 is hard. The reasons for each of the 789 fall into a myriad of reasons that we have only begun to touch on.
What I am going to do now, is to give this old tool a serious look to see how I might be able to develop patterns and algorithms that can automate some of this matching.
As an exercise for any glutton of punishment, I dare you to download the Excel spreadsheet for both Clements 2022 and IOC 2023; combine them into one spreadsheet with Clements Things on the left and IOC Things on the right; then go through all 31,294/30,907 Things and attempt to match them. Call me back in a year from the insane asylum and let me know how you are doing. I say this in jest. Please don't try, it will drive you crazy.
IOC, to their credit, tends to produce just such a spread sheet each year. Their latest compares Clements 2022 to IOC 11.2 2022. And it only deals with the species. No attempt whatsoever to align the subspecies. And, I have a tool (yep, you have no idea how much time I have spent trying to make the best possible alignments) that analyzes that spreadsheet. It contains 8-10% errors approximately. So, how do you think the IOC team fared at this task? And they only deal with the species!
So, I will leave it here for now and take a look at improving this tool that finds all mismatches and see what I can come up with. I have already fixed the Mouse-colored Tyrannulet split situation using it. But that was easy. Stay tuned and let me know your thoughts and ideas.
As to the first problem of avoiding the initial loss of a sighting to a taxonomy. The Reconcile report as given now does not effectively do this. It gives list of things lost, and things gained, but not the correspondences betweeen them. A very useful addition would be to either give all transitions from one Thing to another,
The current reconcile report does that. It is at the bottom and shows all assignments made by the user. Yours shows Zero. Indicating you did not assign anything yet during the reconciliation of IOC.
Here is what mine shows in a fresh test database that contains Clements 2021 and 2022 and IOC 11.2 and 13.1.
In that fresh database I added 4 sightings of Eastern Meadowlark using Clements 2021.
Then I Reconciled Clements 2021 against 2022 and assigned two to Eastern and two to Chihuahuan Meadowlarks.
Then I ran the report and here is what I got. And it does show correctly every action I took.
Now, after that, if I try to reconcile IOC 11.2 against 13.1, there is nothing to reconcile. And the report shows Zero things reconciled, as in your case, because with IOC, I did not reconcile any sightings.
Now, you should run the same report by specifying Clements 2021 and 2022 in the Reconcile window and clicking the Report button. There you should see all changes you made.
or just to flag the instances where one Thing in the old taxonomy has no corresponding Thing in the new taxonomy.
That is precisely what the Reconcile window is showing! The checkbox in the lower left-hand corner is checked by default. Thereby only showing those Things with sightings where one Thing in the old taxonomy has no corresponding Thing in the new taxonomy. Only the Things you have to deal with.
If you want to see the complete list, what you are asking for, uncheck that box and click Analyze again.
Second, it would be great to have a tool to run that would look at all the sightings in the database and flag all those that do not have a Thing that corresponds to them in the taxonomy the user enters. Call it the Sighting-Thing mismatch tool.
Nice idea. I have that in the Orphaned Sightings feature. But currently, it flags any sightings that are not classified by any licensed taxonomic list in your database. Truly orphaned.
I see a problem with your idea however, and not sure how to get it to work as expected. For example, I have sightings for plants, butterflies, insects, spiders, mammals, etc. etc. in my database. So, if I asked for a list of all sightings not classified by IOC 13.1 (for example), I would get a lot of irrelevant sightings - a lot. Off the top of my head, I can't see how to make such a tool work as you are wanting it to.
Then there is the unfortunate case of the Meadowlarks where I have literally hundreds of sightings that were lost in my first IOC 13.1 update. This is far too many to expect to reenter.
I don't understand this. If you restored to a pre-IOC13.1 database, and imported the latest IOC 13.1 taxonomy again, then all of your Clements 2022 sightings of both species will show up under IOC 13.1. I just tested this and it indeed worked like this. What am I missing? I only had to reconcile them once, using Clements, and then when Reconciling IOC, that was one Thing that was not needed to Reconcile. The IOC Reconcile report correctly shows that the user changed nothing, but that they gained the Chihuahuan Meadowlark. The Clements Report shows the assignment made and also that they gained Chihuahuan Meadowlark.
The more complex version of a resolution mechanism is that at the end of reconciliation, or upon running the sighting -Thing mismatch tool, the user would be presented with a list of sightings with their original names (or last known Things that matched them), and prompt the user to enter the appropriate name in the taxonomy they are missing from. Again, the burden woud be on the user to figure out which Forest Robin, matches which Forest Robin split.
Yep, that is the reassignment tool I was thinking about. Still thinking.
Let me know what you think after reading this one, and the others.
Sorry my responses are so long, but they require specifics, because as you can see, this is a very complicated topic.
I very much appreciate you willingness to go down this rat hole with me. It gets lonely in that rat hole all by myself! 🙂
@jeff Remember that weird behavior I told you about, where BD kept all my old reconciliations even though I went back to the original pre-IOC 13.1 list? The latest Recociliation Report I sent had no reconciliations because I had done them on previous attempts. The result was that I did not have any reconciliations to notice.
However, when I go back to the original Reconciliation Report I sent with my initial post, I can see that positive assignments are in the Report, but the failure to assign sightings are not. So the Report does not have this critical set of sightings for which no Thing was assigned. I lost 24 things, but it only shows reassignments of the 17 whose observations WERE assigned. The 7 things that were not assigned any new things, including the infamous Eastern Meadowlark are not shown. Adding those to the Report would expose the problem. In addition it would be great to identify the thiings lost completely in the initial list of Things lost.
You said
"I see a problem with your idea however, and not sure how to get it to work as expected. For example, I have sightings for plants, butterflies, insects, spiders, mammals, etc. etc. in my database. So, if I asked for a list of all sightings not classified by IOC 13.1 (for example), I would get a lot of irrelevant sightings - a lot. Off the top of my head, I can't see how to make such a tool work as you are wanting it to."
The query would be all sightings in BIRD taxonomy Y that do not have a corresponding sighting in BIRD taxonomy X. Surely that is possible.
An idea for harnessing the user to keep all taxonomies reconciled
I see that the issue of how to take two current lists and find ALL the correct correspondences is insoluble, as the boundaries of each IOC THING Clements THING may in fact not coincide. My idea for a solution here is to make these IOC vs. Clements comparisons a task that is done only once for each sighting. In most cases this is easy, but when the hard cases arise (Forest Robins and Mouse-colored Tyrranulets), Birder’s Diary leads the user to them, and it is the user’s responsibility for getting these right, just as you now do for the reconciliation of taxonomies now.
To implement this idea Birder’s Diary needs to keep track of two things: The first is all the things that a user has assigned to a given observation over the years – call this the observation tree. The second is the tree of things that each taxonomy has used over the years. I suspect that you already have this.
In the observation tree with parent assignments of things giving rise to child thing. That is, each sighting would have a thing assigned in each taxonomy in use, and the entire sequence of parent things would be retained in the record for that observation, so that one could always see what things are parents of which current descendant all the way back to the initial designation of the sighting as being species X, in taxonomy Y, back in the past.
Here is a simple example of an Eastern Meadowlark sighting in Arizona in February 2010.
The user assigns the name in Clements 2010, and then the name gets rolled over from year to year until 2016. In 2017, the user decides to start using IOC, and reconciles the current descendant Eastern Meadowlark Clements 2016 to Eastern Meadowlark IOC 2017. The parent of this designation is Eastern Meadowlark Clements 2016, and the child is Eastern Meadowlark IOC 2017. Now the original sighting has two things associated with it -Eastern Meadowlark Clements 2016 AND Eastern Meadowlark IOC 2017. Then, in 2022, the current descendant of Eastern Meadowlark Clements 2016, Eastern Meadowlark Clements 2021 gives rise to a new child Thing Chihuahuan Meadowlark Clements 2022. Meanwhile in a separate branch of the tree, Eastern Meadowlark IOC 2017 has given rise to descendant Eastern Meadowlark IOC 2021 11.2. In 2023, this is the parent of Chihuahuan Meadowlark IOC 13.1 2023.
Now there is a second case to worry about, and that is new observations made after the user does the big conversion from Clements to IOC, in my example in 2017. So now let’s imagine the case where the same birder who did the conversion sees an Eastern Meadowlark in Mississippi in 2021, and uses the IOC list to associate it with the THING Eastern Meadowlark IOC 2021 11.2. Now what species is that in Clements? For this there needs to be a way to trace back all the THINGS in Clements that could correspond to the Eastern Meadowlark IOC 2021 11.2. One solution is the same information you already do to convert one taxonomy into another.
In drawing correspondences there are three cases: the correspondence is unique, so no problem; the sighting is immediately assigned a known child in the taxonomy tree for the other taxonomy, for a simple 1:1 example Western Meadowlark IOC 2021 11.2 with child Western Meadowlark Clements 2022. In the second case there is more than one known THING in Clements it could correspond to the IOC thing, in this case could be Chihuahuan or Eastern. The third case is the Forest Robin case where there is no identified correspondence. For the latter two cases, the user would AT SOME POINT have to make that correspondence and state that this new sighting Eastern Meadowlark IOC 2021 11.2 corresponds to Eastern Meadowlark Clements 2022. It would be very annoying to trigger this reconciliation every time a new sighting is entered, so the thing to do might be to assign it a tree of Eastern Meadowlark IOC 2021 11.2, child unknown CLEMENTS 2022. Whenever the user asks for Clements observations in a particular date range or geographic location, sightings BD would look for sightings in that range with the unknown child designation on the Clements side. If any were found, the user would then be asked if they want to do a reconciliation for the XX sightings involved. This would also be triggered when the user wants to reconcile Clements 2022 with Clements 2023.
The beauty of this tree representation is that no direct comparison needs to be made between the IOC branch of the tree and the Clements branch, except when the user chooses to do so. The user might get the correspondence wrong, just as they can do in the current reconciliation of two taxonomies, but that is not your problem. You made it possible to do it right, and that is all you can do. Another advantage of keeping the tree representation in the database is that it will be possible for the user to see the decision they made when they reconciled an IOC and a Clements observation, and change it if they decide they had made an error.
@jdhoule01 I appreciate the time you put into this and the details you provided.
An idea for harnessing the user to keep all taxonomies reconciled
I see that the issue of how to take two current lists and find ALL the correct correspondences is insoluble, as the boundaries of each IOC THING...
I have thought about this post a good deal and considered benefits and potential paths for more issues. I think this is fraught with a mess of more issues. I don't currently maintain data to the point that this idea would require.
I think that providing a tool simply for IOC v. Clements that would show the Sightings mapped by one, but missing from the other (both sides) and allowing the user to adjust as necessary is the best way forward. It would not work for anything other than comparing the latest IOC to the latest Clements. Any other comparison would be invalid and give false information.
At least initially, I don't think this tool will do anything other than to highlight such sightings. No action buttons or anything else; simply list - for each of the 2 taxonomic lists - the sightings for Things in one taxonomic list that are not in the other.
After I release this and you play with it for a while, it might become obvious that it needs a tweak or action, etc to be added. And we can address that then. But my guess is that just seeing the list will be enough to use the other tools (Sightings Editing and Sightings Entry) to correct issues.
In addition, there are the following existing tools that also can be utilized for edge cases and more information:
- Reconcile Wizard: Using the 'Only Show Things with Sightings' checkbox here will let you know all things in a previous version of a taxonomic list, not present in the new version.
In addition, you can reconcile the latest version of any tax list against any previous version of the same tax list (e.g. IOC 13.1 vs IOC 10.2) to see if you left any behind. If you no longer have the previous versions of a tax list, then these sightings are found by using the next feature: Orphaned Sightings. - Orphaned Sightings: This feature will find any Sightings you have that are not classified by any licensed taxonomic lists in your database. So, by removing older versions of IOC and Clements (Features | Advanced | Remove Unused Taxonomic Lists), this tool will quickly show you which sightings got left behind during a previous reconcile or other issue.
I have uncovered a potential problem which I was afraid of that I think occurred after I ran the 'Match IOC Things to Clements Things' function on the last release of IOC 13.1 2023. It seems that some tagging information (used by the Reconcile wizard) was lost. Not sure yet exactly where that data got lost at. So I need more testing and debugging in that area.
So, the plan now is to attack this intractable issue iteratively. We will first throw this new tool at it that will shine a light on Sightings in one missing from the other. And move forward from there with testing.
Please let me know if you see any problems with this or have any ideas on improvements for this plan. I am starting now.
The query would be all sightings in BIRD taxonomy Y that do not have a corresponding sighting in BIRD taxonomy X. Surely that is possible.
This is exactly what I am working on now. Just remove the word BIRD. And you can compare any two taxonomies. Of course, it would only make sense in the case of Clements 2022 and IOC 2023. It wouldn't make sense for IOC 2023 vs IOC 2022; that is what the Reconcile wizard is for. It would only be for current Clements vs current IOC. This is a very specific tool to solve only this one issue. As that issue does not exist elsewhere anywhere in BD. Only in Clements vs IOC do we have this issue. And only when users attempt use both simultaneously.
I spent 2 hours just now on the first 20 or so of the 789 mismatches, and the reasons are all over the board. Splits/lumps/additions/deletions recognized by one but not the other. Inconsistent or errant classification naming. Mismatched or errant common names. On and on.
But having the user focus on those things that would affect them is the way to go.
About to kick off for the day. Need a stiff drink. Back to this tomorrow.
I have uncovered a potential problem which I was afraid of that I think occurred after I ran the 'Match IOC Things to Clements Things' function on the last release of IOC 13.1 2023. It seems that some tagging information (used by the Reconcile wizard) was lost. Not sure yet exactly where that data got lost at. So I need more testing and debugging in that area.
Update on this. I found the problem and it has nothing to do with the IOC-2-Clements matching. It was only how I creating the test database for this.
I started with a fresh DB with only Clements 2022 and IOC 2023 in it. Then imported Clements 2021 and IOC 2021. This import is what corrupted the taxonomy tagging used by the Reconcile Wizard. Of course, no customer is going to import the tax lists in this reverse order and attempt to reconcile sightings. Once I imported all 4 tax lists in the correct order, everything was correct.
Pheewwwww. That was causing me loss of sleep last night. Glad it was a simple no-problem issue.
fyi only
Dear Jeff,
The new tools you describe should be a reasonable way to address the fears about losing older sighthings. I will be happy to try them out and report back.
I understand if my ideas about taking the burden of IOC vs. Clements out of your hands as much as possible are not practical to implement. Still, if there are other ways of minimiizing the effects of Clements updates on IOC lists, and vice versa, that might help you. The problems we have discovered with this IOC 13.1 update seem to have arisen when the Clements 2022 had an effect on the IOC 11.2 Things.
I hope that you can also augment the Reconciliation Reports for future updates to flag the loss of sighting identities.
Also, I have deleted some older taxonomies whose updates might also have caused undetected problems. The worst case is I guess where a sightng is lost first in one taxonomy, and then again in other at the next update. And the orphan sightings would catch that I guess?
Hi David,
Again, thanks for all your help, input and critical thinking on this matter.
The problems we have discovered with this IOC 13.1 update seem to have arisen when the Clements 2022 had an effect on the IOC 11.2 Things.
I am not sure I see what you mean here. If the Clements 2022 import/reconcile had an effect on IOC 11.2 Things, that would be a bug. That should not be possible. I leave all older sightings in place, such that everything IOC 11.2 used to be able to see, should all still be exactly as it were. If you find different, please let me know.
I hope that you can also augment the Reconciliation Reports for future updates to flag the loss of sighting identities.
Not sure how that would work. This new tool is something you would run after an import/reconcile of either a new IOC or new Clements taxonomic list. And it will show you sightings that don't line up between the two latest versions in your database. With that said, you can see how you might get into trouble with fine-tuning after each update when, for example, IOC 2024 is not supposed to align with Clements 2023 and vice versa. But, we will leave that up to the user.
What did you have in mind with this statement?
Also, I have deleted some older taxonomies whose updates might also have caused undetected problems. The worst case is I guess where a sightng is lost first in one taxonomy, and then again in other at the next update. And the orphan sightings would catch that I guess?
Yes, that is correct. And it allows you to edit the sightings and take corrective actions to properly place the sighting in one or more tax lists.
I should have the new feature ready for download and test later today or tomorrow.
