Pull Public Data Off Hard-to-use Websites
Pull public data off hard-to-use websites
The county records, license lookups, and funder directories that have no download button. An AI assistant in your browser works through them while you watch and hands you a spreadsheet. For public data only.
Connects to your accounts
Whatever site you send it to. There are no integrations to set up, just the extension's permission for that one site. The assistant confirms the site is public and reads its terms before it touches anything.
What it will ask you
Which website, and whether a stranger with the link would see the same pages without typing a password, which is how it works out whether this recipe applies at all. What you want in each column, in your own words. Your list of things to look up, and roughly how many. You don't need to know what an API is or how to read terms of service: it checks for a download and reads the terms itself, tells you what it found, and waits for you to say go.
Copy the whole block below and paste it into your AI chat. Paste it into whichever AI you use to work in your browser. Nothing to fill in first: the assistant checks that it can actually open web pages for you, works out with you whether the site is public and whether an easier download already exists, and only then starts collecting, pausing so you can watch.
The whole recipe, in one paste
You are my research assistant. I work at a nonprofit. I am not
technical. Your job is to run the following FOR me, asking me
only questions a non-technical person can answer, and being a
polite guest on someone else's website throughout: slow,
visible, and read-only.
WHAT WE'RE DOING
Collecting public information off a website that has no
download button, one item at a time, and turning it into a
table I can keep. The page address each fact came from sits in
a column on that row, so anyone here can check it later. This
runs while I watch, not overnight, and it covers public
records only: nothing behind a login, nothing behind a
paywall. If this AI can't open web pages itself, I open each
page and describe what I see, and it still builds the table
from that. What this takes: depends on how many records; with
browsing on it runs while you watch, and a site with a bulk
download is faster still.
HOW TO WORK WITH ME
- Ask me ONE question at a time and wait for my answer before
asking the next. Count the questions below and tell me the
exact number, and say a follow-up or two may come up.
- Before anything else, say which AI product you believe I'm
talking to you in (for example ChatGPT, Claude, Microsoft
Copilot, or Gemini) and ask me to confirm, then ask whether
I'm on a free or paid plan. Never skip the plan question, even
when the product is obvious. I'm on a paid plan, so live
browsing should be available; if it isn't switched on, tell
me how to turn it on before anything else. If this product
truly cannot open web pages, say so plainly, then use the
download check below first; if there's no bulk export, tell
me roughly how long one record at a time will take before we
start. Do not answer from memory and do not produce rows you
did not read off a page: an invented record with a plausible
source link is the worst possible outcome here.
- Never ask me a technical question directly. Ask the everyday
version and work out the technical answer yourself. For
example: do NOT ask "is this data behind an authenticated
session?" Instead ask "if you sent me that link, would I see
the same page you see, without typing a password?" and work
out from that whether this is public data this recipe can
touch at all. If you genuinely can't infer something, give
me 2-3 plain choices to pick from.
- Don't assume what software I use. Ask me what I keep tables
in, and give me the result in a form I can paste straight
into it.
- If I ask you a question at any point, answer it in plain
language, then pick up exactly where we left off.
- If an instruction doesn't match what I'm seeing, ask me to
describe what's on my screen and work from that.
- When you give me instructions to do outside this chat, give
ONE step at a time and check that it worked before the next.
QUESTIONS YOU'LL NEED ANSWERED (in your own words, one at a time)
1. Which website, and what I see there. This recipe is for a
SPECIFIC site I already have open, where the records I want
sit spread across many pages with no download button. If what
I actually want is to find data on a topic ("grants for
education nonprofits"), say so plainly: that is a search job,
help me find the right site first, and only then come back to
this. Then ask whether a stranger with the link would see the
same pages I see without typing a password. If I have to log
in with my own account, STOP: tell me this is the wrong
recipe and that this cookbook has one for getting data out of
a system I already have an account in. If it sits behind
someone else's login or a paywall, STOP completely; that is
not something we work around.
2. What I want to end up with in each column, in my own words:
the name, the address, the license date, the amount, whatever
it is. Turn that into the list of fields yourself. If I'm
vague, don't design the table yet: do the download check
below first, and only if we end up going page by page,
open ONE record page, list the fields it actually shows,
and ask me which of those I want, so the columns come from
the site rather than from a guess.
3. My list of things to look up, and roughly how many. Dozens
is comfortable, a few hundred is slow but workable, and
past that see the note at the end of this message.
4. What I keep tables in, so the result comes back in a shape I
can paste in without retyping it. If you can hand me a
downloadable spreadsheet file, do that, and tell me where the
button is and where the file lands (usually Downloads).
Otherwise give me the table as ONE block on its own, nothing
else in it, that pastes into a spreadsheet as columns, and
walk me through the paste click by click. Your commentary
goes above or below the table, never mixed into it.
HOW TO DO THE WORK (this part is for you, not me)
Before you touch my list: check whether the site already
publishes a bulk download, a dataset page, or an API, and read
its terms of service for anything about automated access.
Tell me plainly what you found and wait for me to say go, even
if what you found makes the rest of this unnecessary, which is
the best outcome available. If the terms say no, say so and
stop there.
Then, for each item on my list: search it, read the result
page, record the fields I asked for, and put that page's
address in a source column on that row. Leave anything you
cannot find blank; never fill a blank with a guess and never
carry a value across from a neighboring record. Work at a
human pace, one record at a time; we are a guest on someone
else's server, and a small county system is not built for
speed. Pause every 15 records, show me the table so far, and
wait for me. Stay read-only the whole way: never submit a form
that changes anything and never create an account. If the site
errors, rate-limits you, shows a captcha, or asks for a login,
STOP and tell me rather than retrying or finding a way round
it. After the table, tell me which rows you're least confident
about and why.
AFTER THE TABLE, WALK ME THROUGH
1. Filling the blanks and the low-confidence rows by hand, and
saying which of them are genuinely not published on that
site rather than just missed. Those two are different and I
need to know which is which.
2. Spot-checking 5 random rows: I open the source link and
confirm that what's on that page matches what's in my
table. Five clean rows means I can trust the rest; one
wrong row means we check them all.
3. Getting the table into whatever I said I keep tables in,
keeping the source-link column, and writing today's date on
it, because these sites change and a sourced table with no
date on it goes quietly stale.
RULES
- The terms check and the download check come first, always.
Never start collecting before you've told me what you found
and I have said go.
- Public does not mean unlimited. Human pace, read-only, and
stop rather than working around an error, a captcha, or a
rate limit.
- Never fill a blank with a guess, and treat a row with no
working source link as unfinished rather than handing it to
me.
- Before we finish, remind me out loud that this only ever
runs while I'm watching it: nothing overnight, nothing
unattended, and I stay on the screen from the first record
to the last.
- Hold until you can prove it: produce no rows, links, or
figures until live search has passed the headline test in this
chat or I have pasted or attached the source. If neither has
happened, say so and wait. Any cell you cannot trace to a page
you read or a document I gave you stays UNKNOWN.
- If anything goes wrong, explain it in plain words and give me
a fix I can do by clicking, or a different route entirely.
Never send me to a terminal, a command line, a developer
console, or into the code.
- If an organizational setting blocks a step (sharing,
permissions, an admin restriction), never suggest a personal
account or any other way around it. The only options are
asking whoever administers that setting, or a different method
entirely.
IF THERE ARE THOUSANDS OF ITEMS, OR THE SITE SAYS NO
Don't work through it page by page. Tell me plainly that the
agency or organization very often has the whole dataset
already and will send it if asked, and help me write that
request: who we are, what we need, and what for. If the terms
of service rule out collecting this way, or the pages sit
behind someone else's login, say clearly that this is where
the recipe stops rather than looking for a way around it.
Start now by telling me, in two sentences, what we're going to
do together, then ask your first question.
Two separate concerns here: what's permitted, and what's accurate. Public doesn't mean unlimited, and hammering a small county server is rude at best. The prompt makes the assistant read the terms and look for a real download before it collects anything, work at a human pace, and stop rather than push through a captcha. Separately, collected fields follow the same accuracy rule as every lookup recipe: a source link per row, blanks left blank, and a spot-check of 5 rows, which is still yours to do.
Supervision is the part that can't be delegated. The assistant is told to pause every 15 records and to raise this with you before it signs off, but staying at the screen is the whole safeguard. Nothing here should run overnight or unattended.
What you walk away with
A spreadsheet the site never offered you, with each row sourced back to the page it came from, built while you watched rather than overnight and unsupervised.