Welcome to the Birder's Diary Forum for Support And General Questions
Use the Support forum for all questions or issues.
Use the Wish List forum to leave your ideas for improving Birder’s Diary.
Use the Community Sharing forum for sharing Photos, Trips, Stories, etc.
Setup your Forum photo and profile here.
You said:
"I am not sure I see what you mean here. If the Clements 2022 import/reconcile had an effect on IOC 11.2 Things, that would be a bug. That should not be possible. I leave all older sightings in place, such that everything IOC 11.2 used to be able to see, should all still be exactly as it were. If you find different, please let me know."
If I look at Eastern Meadowlark in IOC 11.2, all sightings are there, and all Chihuahuan Meadowlarks are still called Eastern Meadowlarks. The update that lost all those Meadowlarks was from IOC11.2 to IOC 13.1, not Clements 2022 to IOC13.1. The IOC13.1 update should have transitioned Eastern Meadowlarks to either Chihuahuan or not, yet the reconciliation tool never showed me those sightings, and indeed they got lost. If Clements had not been updated after 11.2 was updated, how is it that all those sightings got lost?
Hey David,
To do the cool quotes like you see below, select some text from my post by clicking and dragging with your mouse. When you have selected the text you want to reference/quote you will see a double-quote button appear. Click it and the selected text will be added to the bottom of the current reply or a new reply if no reply has been started.
If I look at Eastern Meadowlark in IOC 11.2, all sightings are there, and all Chihuahuan Meadowlarks are still called Eastern Meadowlarks. The update that lost all those Meadowlarks was from IOC11.2 to IOC 13.1, not Clements 2022 to IOC13.1. The IOC13.1 update should have transitioned Eastern Meadowlarks to either Chihuahuan or not, yet the reconciliation tool never showed me those sightings, and indeed they got lost. If Clements had not been updated after 11.2 was updated, how is it that all those sightings got lost?
Here's the explanation of exactly what happened:
- IOC 11.2 stays as it was without any issues after a Clements 2022 import and Reconcile. No change.
- After reconciliation, all older tax lists view of the Sightings remains unchanged. This is on purpose.
- When Clements 2022 was reconciled, you moved some E. Meadowlark sightings to Chihuahuan Meadowlarks. All of those E. Meadowlark sightings were marked as Reconciled.
- When you imported IOC 2023, its Chihuahuan Meadowlarks were erroneously not assigned to the same Thing as Clements 2022.
- When you went to Reconcile IOC 2023, those E. Meadowlark sightings were already marked as reconciled, per 3rd bullet above. So they were no longer presented as needing reconciling. This is correct. And everything would have been fine, if Chihuahuan Meadowlarks were correctly aligned across the two tax lists. This is one of the things I corrected with the 2nd release of IOC after you reported this; the Chihuahuan Meadowlarks now point to the same thing.
- The Reconcile is working as designed. It was the mismatch between the two things that caused the issue. This new tool will highlight those issues so that we can correctly identify and deal with them.
That is the explanation and the plan to deal with it.
Let me know if I missed something.
@jeff If that is the case, your description does not match the behavior of the original reconciliation. BOTH the Eastern and Chihuahuan Meadowlarks were originally lost moving from IOC 11.2 to 13.1, not just the renamed Chihuahuan.
Why would being marked Reconciled moving from Clements 2021 to Clements 2022 affect a reconciliation from IOC11.2 to IOC13.1? That makes no sense.
I realize that you have modified the IOC 13.1 data to properly handle the Meadowlark case now, but I am assuming that the underlying code that allows the problem is still in place, and this would still affect those edge cases not yet built into the new taxonomy data.
@jeff If that is the case, your description does not match the behavior of the original reconciliation. BOTH the Eastern and Chihuahuan Meadowlarks were originally lost moving from IOC 11.2 to 13.1, not just the renamed Chihuahuan.
Hi David,
In the middle of a long post about the new feature/tool. Saw this post and want to reply.
Yes, both Eastern and Chihuahuan sightings were lost when viewing sightings from IOC 13.1, after the reconcile from Clements 2021 to 2022.
The reason is that both of these Things were new Things in IOC 13.1 and didn't match Things in the older IOC 11.2 nor the Clements 2022. So you don't see any of those sightings. Completely understandable and expected because I hadn't properly sync'd the things between IOC 13.1 and Clements 2022. If you were only using one or the other taxonomic list, you would never see this issue and everything would work perfectly. Then, when I produced the updated list, IOC 13.1 now had the same Things for Eastern and Chihuahuan as Clements 2022, and it all lined up.
If I stated that incorrectly before, I hope this clears it up.
Why would being marked Reconciled moving from Clements 2021 to Clements 2022 affect a reconciliation from IOC11.2 to IOC13.1? That makes no sense.
Good question. And one I had to think about. I am suffering from Stack Overflow at the moment. But I have a good answer. LOL.
The Reconcile window shows All things you have sightings for in the older taxonomic list that are not in the new taxonomic list, omitting those sightings for Things that have already been reconciled. Why? Because if you didn't, then those sightings for those Things would keep showing up in your Reconcile list and you would never get to the bottom.
This works perfectly and you would never question this, except for only in the case of using both IOC and Clements.
But, even in that case, when the alignment is correct, it is the right thing. Once IOC 13.1 Eastern and Chihuahuan were correctly matched to Clements 2022 Things, you only needed to Reconcile those sightings once, either with Clements or IOC, and all would be correct. This is working correctly and how it should work; except when these edge cases pop up.
I realize that you have modified the IOC 13.1 data to properly handle the Meadowlark case now, but I am assuming that the underlying code that allows the problem is still in place, and this would still affect those edge cases not yet built into the new taxonomy data.
Well, yes and no. The Reconcile is working properly. The problem is, and always has been and always will be, correctly aligning Things between the two tax lists.
It is my hope that this new tool/feature will help with that.
What it will certainly do, is to point out where all of any user's edge cases are, if that user is attempting to use both IOC and Clements. Now, wait for my new post to be completed (warning: it is long) to see what I find there.
Thanks so much for taking the time to stay in this Rat Hole with me!!! Anyone else want to jump in? The water is fine; a bit muddy; bud fine. LOL.
All righty. Here is what I have. This is detailed and rather long, it is a complicated issue and you can't do it justice unless you dot all the i's and cross all the t's. I do my best to explain and document everything below. Please don't feel obliged to respond unless you want to and have something to add. Welcome to the RAT HOLE. 😉
Creating a proper test for most of the edge cases is what I have done below. I am merely trying here to document this process and the issues I find and potential solutions.
I have the tool working that shows Sightings attached to Things defined in Clements wherein that Thing is not defined in IOC; and vice versa.
As you can see, I have populated an otherwise empty database with some sample sightings demonstrating the issue from both sides.
- I first imported Clements 2021, then IOC 11.2.
- I then added sightings for 3 Forest Robins using Clements 2021.
- I then imported Clements 2022 and IOC 13.1.
- I reconciled Clements 2022 against Clements 2021; I assigned one of each of the three sightings to S. mabirae, S. erythrothorax and S. pyrrholaemus. It was a 3-way split in Clements, but no split in IOC.
- Then I added one sighting of S. pyrrholaemus using Clements 2022.
- I then added 2 sightings of American Barn Owl and 1 of African Grey Woodpecker centralis using IOC 13.1. Because I know that Clements 2022 does not have these Things classified in the current versions of these tax lists.
- Reconciling IOC 13.1 against 11.2 correctly shows nothing to reconcile. As nothing changed between the two tax lists for the the sightings mentioned.
The view of all sightings using Clements 2021. This view is correct. It cannot see the 2 things entered with IOC as they are not defined.
The view of all sightings using Clements 2022. This view is correct. Forest Robin sightings reconciled as shown.
The view of all sightings using IOC 11.2. This is correct. Even though the Barn Owl and woodpecker sightings were entered using IOC 13.1, they are pointing to the same Things in use in 11.2. It sees the Forest Robin sightings (all 4). The first 3 were entered with Clements 2021 which shared the same mapping with IOC 11.2. One of the Reconciled things (new monotypic species) is matched to the older subspecies and shows as such.
The view of all sightings using IOC 13.1. This is correct. It is missing the S. mabirae sighting. That was a polytypic split (a new species with multiple subspecies). And the old S. erythrothorax sanghensis is now correctly pointing to the new S. mabirae sanghensis ssp; not the species S. mabirae. If I had reconciled that 2021 S. erythrothorax to S. mabirae sanghensis subspecies, then you would see it here, S. e. sanghensis is defined by IOC 13.1
Now, let's take each of these issues one at a time and discuss the solution.
ONE OF YOUR FORMER FOREST ROBIN SIGHTINGS IS MISSING FROM IOC 13.1
What you are seeing in the screenshot below is the window of the main tool that handles splits/lumps/deletions/additions and tagging for Reconcile.
The left side is how Clements 2021 defined Forest Robin. And on the right you see how it is now defined after the 3-way split.
The "<>" symbol indicates Things are not in the other tax list. The "=" indicates they are present in both.
The highlighted rows indicate that the old Forest Robin species tagged to each of the three new species: S. pyrrholaemus, S. erythrothorax and S. mabirae.
You can see that all the ssp on the left have an "=" sign, meaning their sightings all properly transferred from 2021 to 2022. And the 2021 ssp S. e. pyrrholaemus uses the same Thing as the 2022 sp S. pyrrholaemus. But otherwise ssp to ssp matching.
Now IOC 11.2 and 13.1 did not split the Forest Robin. And all of their ssp auto transfer over to Clements 2022.
IOC 11.2 still shows all 4 Forest Robin sightings. IOC 13.1 did also in the first release. But then I ran the auto-match on SciName to catch the Eastern Meadowlark, and other similar matches. But in the case of the Forest Robin, it broke, because now I am pointing the IOC 13.1 S. erythrothorax to the Clements 2022 S. erythrothorax. So, it cannot see the older sightings attached to the older Thing no longer in IOC 13.1 and Clements 2022.
So, while the auto-match-on-SciName helped in one case, it broke another case. RAT HOLE.
The solution for this one is to transfer your Clements 2022 S. mabirae sightings to S. m. sanghensis. That Thing is defined by both taxonomic lists. That is just in this specific case.
IOC SIGHTINGS FOR AMERICAN BARN OWL AT NOT IN CLEMENTS 2022
This is a reverse of the Forest Robin situation. IOC did a 3-way split with Barn Owl, Clements did not.
Due to the matching routine run for the 2nd Ed. of IOC 13.1, Clements 2022 Barn Owl, Tyto alba was matched to IOC 13.1 Western Barn Owl, Tyto alba. It should not be matched to that.
The solution for this one, again, would be to move each of these two sightings to the appropriate subspecies. The subspecies for each of the three species of 'Barn Owl' in IOC 13.1/11.2 use the same Thing as the subspecies of the Clements 2022/2021 Barn Owl.
IOC HAS SIGHTINGS FOR SSP AFRICAN GREY WOODPECKER CENTRALIS BUT NOT CLEMENTS 2022
This is because Clements 2022 does not recognize this taxon.
The solution for this one might be to move the sighting up to the species level, where the two tax lists agree, and note in the comments field of the sighting the reason for the move.
------------------------------
All right, that is enough for now. If you don't need a Scotch yet, then I will drink one for both of us.
Takeaways:
- There will always be problems in the correct alignment of these two tax lists. Just ask the guys at IOC that never even bother with the subspecies, and Cornell that never bothers at all.
- This tool will definitely shine the light on where you might have issues, if any, when attempting to use and keep sightings aligned between Clements and IOC.
Let me know what you think.
Update: Build 28 of v6.0 has this new feature in it. It is now available for download and install. Help | Internet: Check for a program update
Here is my use of the tool in my database. I do not use the IOC taxonomic list for anything other than reference. Clements is my master.
Here is what your database from 2021 looks like when comparing IOC 11.1 with Clements 2021. A copy I had from the last time you uploaded your DB to me.
It appears that most all of these involve splits. And I don't have a solution for them. See previous post where I discuss 3 common issues and the possible solution for them.
The best advice I can give now after reliving all of this for the past few days, is to start recording and even change existing sightings to the correct subspecies in both lists. The subspecies generally line up 98% of the time across the tax lists. That will almost completely eliminate this issue if you want to maintain sightings in both tax lists.
Let me know.
@jeff I want to try the new tool. Upgraded to the latest posted version, build 28, but can't find this tool in the menus. Can you tell me how to find and invoke it?
@jdhoule01 Hi David.
Sorry for the confusion. See first screenshot in my previous post.
Looking forward to your feedback.
I have tried the new tool, and have some questions.
First, When I compare IOC 13.1 to Clements 2022, I get a modest number of mismatches (3 in IOC, and 14 in Clements, not in IOC). When I invoke the reconcile tool between IOC and Clements in each direction, almost all of these appear as changes to be reconciled. I have not done these reconciliations before, as I was assuming that the best reconcilaitions were IOC to IOC and Clements to Clements. Should I just do these cross-list reconciliations?
Third, I was surprised that the tool does not alow comparisons of IOC with IOC, or Clements with Clements. I thought this should be there, as it was going from IOC11.2 to 13.1 that led me to notice the problem in the first place. Why is this not there?
When I repeat the comparison of IOC11.1 and Clements 2021, like you did above, I get the much larger number of mismatches that you found. I have checked a few easy cases with few sightings, and those sightings seem to have reappeared in the current IOC 13.1 and Clements 2022. I do not understand how they come back into alignment. From your past description it sounded like once there was a mismatch, there was no fixing it, but I guess that must be wrong? This has been my main concern, that a mismatch, once made, would escape reconciliation from that time forward.
Next, I do not find an Orphaned sightings, which is good. But what is the Orphaned sighthings defintion? Could I have sightings that are assigned to a Thing in an old taxonomy, but not in one of current ones? If I did, would it register as orphaned?
Finally, you posted this picture
of what looks like a very useful comparison visualization. Can I generate such a chart?
Hi David,
Glad you got to this while it is still fresh in our minds.
When I invoke the reconcile tool between IOC and Clements in each direction, almost all of these appear as changes to be reconciled. I have not done these reconciliations before, as I was assuming that the best reconcilaitions were IOC to IOC and Clements to Clements. Should I just do these cross-list reconciliations?
No. Do not reconcile across tax lists. That would not result in proper functioning.
Third, I was surprised that the tool does not alow comparisons of IOC with IOC, or Clements with Clements. I thought this should be there, as it was going from IOC11.2 to 13.1 that led me to notice the problem in the first place. Why is this not there?
I think there is a fundamental misunderstanding of the problem here.
Let me restate this:
- If you only use either IOC or Clements, and Reconciled older versions to newer versions of each, everything works perfectly as expected.
- If you attempt to use both, the Reconcile still works correctly when going from older to newer versions of the same tax list.
- The problem comes solely from the fact that IOC does not align with Clements. And new sightings carry forward in each tax list in a different branch against Things found in one taxonomic list that are not used in the other. In a perfect World, all the underlying Things will line up.
The tool is created to show you which Things you have Sightings for in the latest IOC that you don't have sightings for in Clements, and vice versa.
The only solution here is to start assigning Sightings in these particular branches of the taxonomy to the subspecies level. 98% of all the subspecies lineup across these two tax lists.
When I repeat the comparison of IOC11.1 and Clements 2021, like you did above, I get the much larger number of mismatches that you found. I have checked a few easy cases with few sightings, and those sightings seem to have reappeared in the current IOC 13.1 and Clements 2022. I do not understand how they come back into alignment. From your past description it sounded like once there was a mismatch, there was no fixing it, but I guess that must be wrong? This has been my main concern, that a mismatch, once made, would escape reconciliation from that time forward.
You are going to have to give me just one case that you are talking about and we will have to look at it and see what happened. I will probably need your latest database so I can follow along on my system with your current data. There are a myriad of paths each of these cases can take. So we have to look at them one at a time.
Next, I do not find an Orphaned sightings, which is good. But what is the Orphaned sighthings defintion? Could I have sightings that are assigned to a Thing in an old taxonomy, but not in one of current ones? If I did, would it register as orphaned?
An orphaned sighting is a sighting for a Thing that is not classified in any licensed taxonomic list in your database.
For example, if you once had the Amphibians tax list in your DB, and then entered a sighting for a Bullfrog, but then later deleted the Amphibians tax list from your DB. That sighting is orphaned. Because it is a sighting for a Thing that is not classified by any licensed tax list in your database.
what looks like a very useful comparison visualization. Can I generate such a chart?
No. That is one of a couple taxonomy building tools I have created. It is not available in Birder's Diary Desktop.
Let me know if you have any further questions. I am kicking off for the night. We could also schedule some time to get onto your computer over the internet and on the phone together to chat about some of these. Let me know.
Third, I was surprised that the tool does not alow comparisons of IOC with IOC, or Clements with Clements. I thought this should be there, as it was going from IOC11.2 to 13.1 that led me to notice the problem in the first place. Why is this not there?
I think there is a fundamental misunderstanding of the problem here.
Let me restate this:
- If you only use either IOC or Clements, and Reconciled older versions to newer versions of each, everything works perfectly as expected.
- If you attempt to use both, the Reconcile still works correctly when going from older to newer versions of the same tax list.
- The problem comes solely from the fact that IOC does not align with Clements. And new sightings carry forward in each tax list in a different branch against Things found in one taxonomic list that are not used in the other. In a perfect World, all the underlying Things will line up.
Well this is exactly why I keep getting confused. I DID an IOC11.2 to IOC13.1 update that lost me seven different species. That is the start of this whole thing, and yet you tell me those updates "work perfectly." It did not work perfectly.
Hi David,
So sorry for all the confusion and frustration this has caused.
I want to answer your questions and then spell out in succinct terms what you can expect from Birder's Diary if you attempt to maintain Sightings in any two comparable taxonomic lists within Birder's Diary. (e.g. IOC and Clements)
Well this is exactly why I keep getting confused. I DID an IOC11.2 to IOC13.1 update that lost me seven different species. That is the start of this whole thing, and yet you tell me those updates "work perfectly." It did not work perfectly.
My statement is, and always has been, that if you only used one tax list or the other then the updates and Reconcile process do work perfectly. The problem comes when attempting to maintain all your Sightings across both. I have also always said, use one as the master for entering sightings and maintaining a life list, and use the other as reference.
I was surprised that the tool does not alow comparisons of IOC with IOC, or Clements with Clements. I thought this should be there, as it was going from IOC11.2 to 13.1 that led me to notice the problem in the first place. Why is this not there?
This new tool was created specifically with the goal in mind of being able to show mismatches between Sightings using IOC and Clements ONLY. The Reconcile Wizard is the tool to show the differences between two different versions of the same taxonomic list.
The Best Way Forward if you want to maintain both IOC and Clements.
- Pick one or the other to be the master. This is the taxonomic list you use to enter sightings, generate reports and Life Counts. Choose IOC or Clements.
- Get the Sightings and Life List in the master taxonomic list looking correct according to how that authority views different taxons.
- When new versions of the master taxonomic list come out, use the Reconcile Wizard against the previous version to correctly deal with splits/lumps/etc.
- When new versions of the non-master taxonomic list come out.
- Do NOT use the Reconcile Wizard against that.
- Instead, use this new tool to compare and make adjustments to the sightings in the non-master list to achieve the best possible matching for your needs. This is easier with a Life List < 1000, but gets tougher and tougher as you get above that number.
- Do NOT make changes to Sightings in the master list to make things agree, with the exception of the next bullet concerning Subspecies.
- Best Option: To make this job much easier on yourself now and going forward, for the sightings that this new tool shows you are out of sync, edit those sightings in the master list and move them to the best possible subspecies. In general, 98% of the time, the subspecies between the two lists will align, and stay aligned going forward.
I think that succinctly covers everything. It gives you the only path forward that I can support.
Let me know how I can help with any of this. Very happy to assist in any way.
I think I would like to do a brief online session with you to try to properly put my sightings back together. The most important thing for me is understand the logic of how to fix things for the future, and working with examples should help. I have some times available during normal working hours next week, or we could find an evening or weekend time if you would be working anyway. I suspect it will not take terribly long, and there is no pressing need for me to fix this quickly.
I also want to point out something in addition raised by your last posts.
My statement is, and always has been, that if you only used one tax list or the other then the updates and Reconcile process do work perfectly. The problem comes when attempting to maintain all your Sightings across both. I have also always said, use one as the master for entering sightings and maintaining a life list, and use the other as reference.
If you attempt to use both, the Reconcile still works correctly when going from older to newer versions of the same tax list.
These two statements are in direct contradiction, hence my confusion.
OK, so I take it you really mean the first point quoted above, not the second.
You may tell people this somewhere, but it was never made clear to me. I have been using IOC and Clements in parallel for quite a few years, and I have never gotten any warning about this.
I can tell you exactly how I got into this situation - I went to download taxonomic lists, and it says click here to get IOC. Nothing on the download page says "If you already use one of these lists, here is what you need to know before you buy another one." I think great, it will be nice to have both, click on it, install it. No warning pops up. I reconcile it with the existing Clements list. No warning ever pops up to say "This will cause problems." Every year I download both the new Clements and IOC lists, and reconcile them with the previous list of the same type There has never been any indication that "Uh Oh, you have already reconciled the XX list, and doing this one may cause problems." When entering a list from a trip where the trip list was kept in IOC taxonomy, I enter sightings usng the IOC taxonomy.
This suggests some relatively easy ways of installing a manhole cover on this open manhole:
- Have the user designate a primary taxonomy when they first use BD.
- Data entry using a non-primary taxonomy is not possible.
- Whenever the user attempts to deploy a second taxonomy, they are warned about pitfalls and referred to documentation on how to do it right.
- When the user attempt to reconcile any non-primary taxonomy, they are warned off and referred to the new diagnostic tool instead, along with instructions on how to use it.
- There is a prominent warning on the taxonomy download page that the user gets before purchasing a second taxonomy.
@jdhoule01 Let's arrange a desktop sharing session via email please. I am available most days, except when out birding or running errands, etc. Just let me know what works for you and we will find a time.
Let me respond to some of the questions/concerns you raise above please.
These two statements are in direct contradiction, hence my confusion.
I can see how it might seem that way without the proper context. This is a complicated issue, and those two statements are factually correct. The issue is not the Reconcile, it is the non-alignment of the taxon between the two taxonomies. The Reconcile does the correct thing in all cases. I recommend however, that in your case, keeping and using both IOC/Clements, that the Reconcile of the non-master tax list is not needed.
You may tell people this somewhere, but it was never made clear to me. I have been using IOC and Clements in parallel for quite a few years, and I have never gotten any warning about this.
On the old site that we had from 2004 until 2021, it stated this clearly. And that site had a lot of clarifications and disclaimers about many things. It just became too much noise, and no one ever read it anyway. So this rebuilt site focused on brevity and clarity; and therefore dropped many of these statements/pages/etc that were simply too much info and no one ever read anyway. This is just one of those cases. I apologize that this one has caused you these issues. It's a constant balancing act of too-much-info vs not-enough-info.
For example, this World has changed so much computer/technology-wise since BD first came out in 1995. The User Guide used to cover every single window and every single button, checkbox, list, etc. on every window. That is never done anymore for any software. Everyone is assumed to know what buttons and checkboxes, etc. do. I rewrote the user guide in the past 2 years to only focus on the top 95% of questions asked. The size went from a couple hundred pages down to a couple dozen.
In summary, just very difficult to get the info balancing act just right all the time. But the way I justify that, is that I surpass all other software entities these days in, when you need that extra 5% help, I am always here; and I own the problem instead of just pushing you off or giving bad advice. I am the 5% solution to this equation. And by the way, this is a 1/5000 user case. So a 0.02% issue. I haven't had anyone else ask about this since IOC was first released and this issue was discussed then. That predates when you first got involved with BD. And hence you not being aware of this previous discussion from years ago.
I like your suggestions about helping the user in this situation (it only exists between IOC and Clements; and no other tax lists that I offer). I will begin looking into how to implement these suggestions. Thanks for taking the time to think about these ideas.
I will be looking for your emails to setup and desktop sharing session.
