Improve robots.txt, hopefully this stops some of the bot spam
Commit 6600b pushed by Anže Pečar

12 tests 12 passed Duration
accounts/tests.py::TestSelectedInstance::test_invalid_selected_instance 1.0166s
accounts/tests.py::TestSelectedInstance::test_no_selected_instance 0.3369s
accounts/tests.py::TestSelectedInstance::test_selected_instance 0.3763s
accounts/tests.py::TestSelectedInstance::test_selected_instance_https_truncate 0.3429s
accounts/tests.py::TestSelectedInstance::test_selected_instance_http_truncate 0.3382s
accounts/tests.py::TestSelectedInstance::test_selected_instance_slash_truncate 0.3445s
accounts/tests.py::TestSelectedInstance::test_selected_instance_through_session 0.5910s
accounts/tests.py::TestSelectedInstance::test_unselected_instance 0.5846s
stats/tests.py::TestStats::test_all_daily_lang_stats 0.0008s
stats/tests.py::TestStats::test_posts_from_yesterday_are_updated 0.0692s
stats/tests.py::TestStats::test_store_daily_stats 0.0604s
stats/tests.py::TestStats::test_store_daily_stats_with_accounts 0.0614s