Cinnamon Hummingbird

Welcome to the Birder's Diary Forum for Support And General Questions

Use the Support forum for all questions or issues.
Use the Wish List forum to leave your ideas for improving Birder’s Diary.
Setup your Forum photo, profile, and signature here.

New Feature Comment...
 
Notifications
Clear all

[Solved] New Feature Comments: World Lifer Report

13 Posts
3 Users
1 Reactions
1,502 Views
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago
[#1881]

I am working on a World Lifer Report requested by a BD user.

The idea is to give the user the data to determine which countries will give them the most lifers. E.g. where should I plan my next trip to get the most Lifers?

This report will compile statistics for each Country in the World, and specify the following stats for each country:

  • Species at Location
  • Species at Location Seen Anywhere
  • Species at Location Seen at Location
  • % of Species Seen at Location
  • Potential World Lifers at Location
  • Endemics at Location
  • Endemics Not Seen at Location
  • Families at Location
  • Families at Location Not Seen Anywhere

It produces this report in CSV file format, so that you can open it with your favorite spreadsheet editor; e.g. Excel.

An example would look like this:

image

In spreadsheet format, you can sort your data by the data in any column, as well as perform other stats on any of the data. For example, sort on which country has the most endemics not seen? Which country has the most potential world lifers for me? 

I have this feature written and working. But looking for comments on what to change/add/etc. before releasing it.

You can also run this report on any 1 country if you select the location instead of leaving that dropdown blank.

Concerns:

  • This report is driven by the existing Checklist data in your database. If such data is out of date or incorrect for any country, it will report erroneous numbers. Tools exist for updating checklist data per location in BD already. Although that would be slow for all countries. I am considering a new tool that will update all countries based on AviBase data online.
  • Takes 4-10 minutes to run on most computers for all countries in the World.

Here is the main window that drives this feature: 

image

Looking forward to some good ideas. Thanks. 


Reply
12 Replies
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

Now - let me show you something really cool. Especially your Excel/spreadsheet aficionados! 

Load this report CSV file into ChatGPT! If you don't know what ChatGPT is yet, what planet are you from. It is the next Internet/Google/Printing-Press that is changing the World. I have been using it ever since it was first publicly released 30-Nov-2022. I use it probably 20-50x per day. Never touch Google again. Enough. 

Load this new CSV report file into ChatGPT, and then ask it for something like:

image

Then I asked it to modify the chart and add number of endemics not seen to it.

 

image

Same thing, but for the bottom 10 potential world lifers.

image

Ok - but only for those where I have at seen at least 1 species at the location.

image

Add another bar for Species at location, seen at location.

image

You start to get the idea. The options are limitless. You talk in plain language to it and it will understand you 99% of the time. And if not, it takes criticism very well.

This of course, is just a sample use of ChatGPT with one of Birder's Diary's latest features. But the uses of ChatGPT are truly limitless. That is not hyperbole. Ask it anything.

ChatGPT is free for 95% of everything you would need it for. https://openai.com

Hope you enjoy!


Reply
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

Update: v6.2 Build 14 just released. It fixes a couple errors with the counts in some cases and adds a few enhancements.


Reply
Posts: 29
(@iasmit)
Eminent Member
Joined: 6 years ago

I just updated to Build 18 and found this new feature. It certainly is useful, providing data direct from Birders' Diary that previously had to be calculated in a separate spreadsheet. Particularly valuable are the "You have Sightings at this Location for Things not linked to this Location" species shown. Some of those shown for my data are omissions from your listings, but others are errors on my part which need to be fixed now that this report has identified them.

Just to mention two problems encountered with it on my desktop - one trivial and one less so.

ScreenHunter 1424

 

The selections made on the input form do not overwrite the instructions in the box - "Select observer for report" etc. A minor bug - still works.

ScreenHunter 1425

The top of my listing for "World", after sorting by "Potential World Lifers" is shown. Relevant and useful data, but surely the figures for endemics are much too low?

Thanks Jeff - Ian

 

 

 


Reply
Jeff
 Jeff
Admin
(@jeff)
Joined: 6 years ago

Creator & Technical Support
Posts: 1440

@iasmit 

Hi Ian,

Thanks for posting. Glad you like the report. It was fun to create. Any many, many folks have found it very useful.

On to your two issues that you found:

Dropdown Hint overwriting typed text
I had one other report of this from another user, but he found that it went away when he closed and reopened the window. If you find that this is consistent for you, please let me know. I would love to get onto your computer and figure out why this is happening. I have 5 computers, and it doesn't happen on any of them.

Endemics Numbers not realistic
So, a little bit about how Endemics are calculated in BD. 
Endemic Count = number of species that are linked to this location or any sub-location that are not linked to any other location.
For example, bird XYZ is linked to Indonesia (via the BD Checklist Editor) and not to any other location outside of Indonesia, makes bird XYZ an Endemic of Indonesia.
Gonna save the discussion about the absurdity of attempting to make such bird-to-location data 100% accurate. It is endlessly long and full of issues. But, suffice to say that BD should be > 95% accurate.
Your database clearly is not. My database says that Indonesia has 463 endemics. Yours says 20. Big difference. And this is simply because of the equation I gave you above. You must have bird XYZ (and 442 others) linked to locations outside of Indonesia.
If I look at the AviBase page here for the Checklist of Birds of Indonesia according to Clements 2024, it says that it has 504 endemics.

ONE SOLUTION:

Re-import the Clements 2024 checklist data manually. The file should still be present in C:\Birder's Diary\Taxonomy Downloads\

image

Run BD and choose File | Import/Export..., Checklists Tab

image

The key here is the checkbox labeled Before Import, Purge All Checklist Records for This Taxonomy. It is fairly self explanatory.

You get a clean checklist load for the World that should be much more accurate.

The downside is that you lose all checklist data (bird-to-location; not your Sightings - 'checklist' has dual ambiguous meanings these days) you created for personal locations such as your backyard, local park, county, etc.

The good news is that it creates a complete import file of all Checklist data in your database for the specified taxonomic list before clearing and starting the new import. So you can get back your old location data by pulling it from this saved file, putting it in a secondary file, and quickly importing it. For example, the saved checklist filename will be TaxonomicChecklistPurged_yyyymmdd_hhmmss.csv. Open that file in NotePad, or your favorite text editor - even Excel. Locate your location you want to save, e.g. "Home". Select and copy all records into a blank file called "home-checklist.csv". Now simply import those records back into your database.

How does this checklist data get into this state that you have? Many, many different ways over the years. Editing or updating yourself. Several years worth of data were always added in a non-destructive way - never delete any data, only add to it. This junk accumulates over time, and splits, and different taxonomic lists.

I just did this exercise with Clements 2024. It took about 10-12 minutes for the Save/Purge/Import to complete. Your time may vary depending on computer/disk/cpu speed. Then, I ran the report for just Indonesia, and the Endemic count is now 514! Pretty close to AviBase's number.

SECOND SOLUTION:

Same as above, but

  • Do not check the Purge... checkbox
  • Select Overwrite Locations, instead of Overwrite Things
image

This one will only overwrite the locations in the Clements Checklist import file, and not touch anything else; such as your personal locations like Home, Local Park, County, etc. But the catch is that it may be some of these personal locations that are causing the problem. For example, if you have a sub-location somewhere outside of Indonesia (e.g. South Africa > Ngundu Park) that has 400 of the Endemics that are only found in Indonesia; then per the Endemic calculation above, they will no longer be counted as Endemics for Indonesia. If you don't have this problem, then you might want to try this method before the first one mentioned further above.

OTHER REASONS FOR THE BAD ENDEMIC COUNT

  1. Bad Location Hierarchy
    If you had Indonesia parented, for example, by South Africa; then South African would have all the species that Indonesia has, and therefore none of the species under Indonesia would be Endemic.

INITIAL ANALYSIS FIRST

Let me know if you find some of your issues. Do some analysis up front. Take a look at the AviBase page referenced above using Clements 2024. Run a Checklist report for Indonesia in the report designer with the ENDEMICS checkbox checked. Which birds are failing to show as Endemic in the report in BD that are shown as Endemic in AviBase - just find one to start with. Now open the Checklist Editor (F9), key in that bird name on the right side of that window (don't use the left side) and load it. What locations show for that bird? 

Report Designer

image

Report

image

Checklist Editor

image

--------------------------------------------

Let's stop here for now. Let me know if you have any questions. I definitely want to know what species your analysis uncovers and what locations they were linked to other than Indonesia.


Reply
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

Update: I figured out the problem with the dropdown hint covering the text. Fixed in Build 19 - soon to be released.


Reply
Posts: 29
(@iasmit)
Eminent Member
Joined: 6 years ago

Thanks Jeff for that detailed analysis, and sorry for the delay in responding.

I was reluctant to try your first solution in case country checklists which had been cleared of vagrant records were overwritten, bringing the vagrants back again. Did try your second solution, but that did not seem to make a difference.

Have just run your "initial analysis" for a random endemic in Indonesia, with results as in the screenshot.

ScreenHunter 1439

Without fully understanding the issues, could the presence of "World" under "Location" be causing the problem?

As a final comment, the issue of endemics is not that important to me as they are defined by political and not biogeographic boundaries. But certainly interested to hear your response!

 


Reply
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

Hi Ian,

Yes. No Thing should be attached to location World in the Checklist Editor. Not sure how that could have ever happened as I don't ship checklist data with that attachment.

Clear all such attachments quickly by following the numbered steps in the screenshot below for the Checklist Editor (F9).

image

If you do want to try my solutions to clean up your Checklist Data in your database, simply make a backup of your database before changing anything. If the results are not want you want, need, expect; then restore the backup and you will be back where you were.

A few things in BD, just a few, depend on accurate and reliable checklist data. This new report is one of those things. Checklist reports is another.

Let me know if I can assist further or if this generates any questions.

Thanks. 


Reply
Posts: 29
(@iasmit)
Eminent Member
Joined: 6 years ago

Thanks for the response. Cannot recall adding WORLD to (most?) checklist records but must have done so somewhere along the line, probably in an attempt to fix some problem. But (after backing up my database) have now removed "World" per your screenshot, and the new listing after sorting by "Potential World Lifers" is:

ScreenHunter 1440

The endemics figures now make sense - I have 456 for Indonesia versus your 463 (before correction), but that's close enough for me. Thanks Jeff.


Reply
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

Thanks for the update Ian. Glad this gets the data and the report back to close-to-correct on the Endemics count.

Take care. 


Reply
Posts: 9
(@rlsiebert)
Active Member
Joined: 6 years ago

Jeff,

Just updated to 6.3 and tried this wonderful report. It will be so very, very useful.

I do have a question about the data that appears in the comments field. I resolved several as incorrect handling of splits along the way. But I also found three species that I definitely saw in a location. Should I share this data with you? I can do so privately if you want.

 

Randy


Reply
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

Hi Randy,

Glad you like the report. Many users have had similar responses. You can thank a BD user and birder from Austria for the idea. A few other users helped test and refine it also.

Please do share the species that you have a concern with; either here or in an email to support@birdersdiary.com. 

Checklist data can be out of date and require updating via one of many methods I have built into Birder's Diary.

I can walk you through appropriate methods once I see what you are referring to.

Thanks. 


Reply
Jeff
Posts: 1440
 Jeff
Admin
Topic starter
(@jeff)
Creator & Technical Support
Joined: 6 years ago

After review, I see that I have an in-depth reply above discussing how to deal with such issues; of course I am assuming you have the checklist data issue I discuss above. I won't know until I see it.

But please look above for my reply to Ian on 4-Apr-2025, and see if this doesn't fit your situation. 

image

Let me know please.


Reply
Share: