North America Procurement Council Procurement News

Why the NAPC Ended Free & Open Access to Its Data

November 15, 2018

To reduce fraud and ensure the integrity of the bidding process, all user accounts on all NAPC portals must now pay a $20 annual authentication fee or become a premium member. Viewing full bid lead reports now requires login.

Due to the continued fraud and violation of the Terms of Usage of the portals administered by the North America Procurement Council, all users (except bonafide government agencies) must now pay at least a $20 registrant authentication fee or join as a premium member. Previously free and unauthenticated accounts can no longer access the full bid lead report.

The North America Procurement Council is a public benefit corporation (social enterprise) division of Bid Ocean that started with the mission of providing free and open-access procurement portals to publish most all government procurement solicitations. It is not a profit-making venture.

We at the NAPC took this task upon ourselves because democracy requires transparency in government procurement and if government won't provide it then it is up to the people to do so. We estimate that improper procurement practices cost the United States more than $500 billion annually. Some of this money is diverted to criminals who then use a portion of their ill-gotten gains to subvert democracy and steal more of taxpayer's money.

Bad procurement fuels the erosion of democracy. We want to live in a democracy that adheres to the values of the U.S. Constitution and the The Universal Declaration of Human Rights, therefore, we support transparent government procurement so that irregularities can be more easily identified and addressed. 

The waste and fraud in goverment spending hurts all of us in many ways. It means bad roads and bridges and that means higher vehicle maintenance costs, more traffic accidents, delays in travel, higher transportation costs and much higher maintenance costs later. It means lower quality education and health-care for our children. It means more police brutality and violation of our rights. It also results in a more polluted environment, which can make us sick and die.

Our tax dollars stolen through government contracting is used to keep criminals in political power. The recent U.S. mid-term elections were the most expensive in history and it is likely that a portion of that money spent persuading us to vote for the next corrupt politician was stolen from us in the first place.

Corruption grows if it isn't opposed.  And if we do nothing, as most of us have, one day we wake up and realize that we no longer live in a democracy, that our roads are terrible, that our health care system is by far the most expensive and worst among developed nations, that schools may do more harm than good and that our environment is polluted and endangering our health. We may realize that instead of relying on our local police to protect us and serve our needs we now have to fear them. Some of us may even realize that more than 25% of children in the U.S. needlessly live in poverty due to bad government.

What some of may not realize is that some countries which are vastly less wealthy and even rank among the poorest on Earth are able to provide good roads, free health care and free primary and higher education. They can do this because they have good government and tax dollars aren't used to enrich a small evil elite but are used to benefit the people. They also don't spend a trillions dollars each year on waging war against manufactured enemies.

Many Americans are doing all they can to shine the light of truth into the dark corners and defend what little democracy and freedom remains, but many more won't do anything and by doing nothing they make things worse for everyone, including themselves.

We didn't want to be those people who do nothing. With the amount of procurement data publicly available actually decreasing and few visible efforts to create statewide public bid portals, we felt that if we wanted to support democracy in the United States that we should create the portals ourselves at no cost to the people or government. We hoped that enough people would appreciate what we were doing and would contribute to the cost by taking a paid subscription.

It was a good idea and in fulfilling our mission perhaps we have made a small difference. However, given the realities of human nature, fulfilling that mission has been challenging and our approach has had to evolve.

Data scrapers and fraudulent user registrations.

Initially we just made all data freely accessible without login, including access to bid documents. That resulted in massive scraping of the portals using software bots, which was in violation in our clearly stated Terms of Usage. The data scrapers weren't just using software to grab new data, they were scraping all data repeatedly. This resulted in substantially greater server costs and overloading of the servers so that the legitimate users couldn't access the data they needed. So, we had no choice but to require registration and login to access bid documents and reduce the amount of older data accessible. The data scrapers simply got more clever, created fake user registrations and scraped a bit more slowly.

Some of the data scrapers are our competitors, who then resell the data that costs us more than $1.5 million annually to acquire and process in the most cost effective way possible. The bots used to scrape our servers are mostly in India, Russia and Europe but many are also in the United States. We keep blocking their IP addresses and they keep changing them in a never-ending game of whack-a-mole. 

Due to the number of fradulent user registrations we had to impose a form of user authentication that would dissuade the worst abusers of our good will. So, we imposed a $10 annual authentication fee to be paid by credit card or Paypal. This reduced the number of fraudulent registrations, but then we found that even that did not stop the fraud and abuse. What some registrants did was to contest the charge they agreed to by doing what is called a credit card charge-back. Some do repeated charge-backs that end up costing us hundreds of dollars in fees and labor costs. Each single charge-back costs us $30 and the policy of most credit card processors is that the merchant is always wrong and any appeal is a lengthy and time-consuming process. 

To compensate for the cost of fradulent charge-backs we have raised the cost of authentication to $20 and restricted foreign registrants.

To further reduce data scraping we have removed the full report from public pages and now require user authentication and login to view the full report. 

These changes have upset some people but others welcome the changes because it helps secure their valued source of data at a negligent cost.

Those interested in purchasing our data in bulk to resell should contact us. The cost is likely far less than scraping it or paying large numbers of people to manually enter the data.