Compare commits

18 Commits

Author SHA1 Message Date
3e64189f8f 2023.1 release notes 2023-01-13 10:33:34 +01:00
45dc173ea7 Merge branch 'feature/3-provide-application-health-check' into 'develop'
Implements #3, a /ping endpoint

See merge request ruben/jail2ban-pf!5
2023-01-13 09:30:03 +00:00
9b85bfabdb Implements #3, a /ping endpoint 2023-01-13 09:30:03 +00:00
969ba0f64c Switch to python 3.9 image 2023-01-09 22:58:27 +01:00
d9b5d36835 Not yet 2023-01-09 22:55:21 +01:00
9f86e143fe Oops vim autoindent 2023-01-09 21:51:10 +00:00
a49da1f3ef Enable SAST 2023-01-09 22:47:34 +01:00
36ff86c71e Looks like the gitlab-ci.yml syntax was changed. adjust
https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscoverage_report
2023-01-09 22:38:14 +01:00
29f6e6093b Missed a spot 2023-01-09 15:01:06 +00:00
ccc7165d1b More documentation fixes
Using the web based markdown editor for this to get right
2023-01-09 14:59:52 +00:00
72f0e095ca Fix code blocks, add additional documentation 2023-01-09 14:50:52 +00:00
61869049a0 Picky markdown... 2023-01-09 15:17:38 +01:00
c868c63aa7 fix documentation verbatim blocks 2023-01-09 15:11:40 +01:00
9875dccec0 For use with the documented uwsgi 2023-01-09 15:08:02 +01:00
359514e581 Update documentation 2023-01-09 15:07:29 +01:00
9ed6b65b6d Exclude software testing reports 2023-01-09 14:50:03 +01:00
1c2e3413ec Merge branch 'feature/01_test_all_the_things' into 'develop'
Feature/01 test all the things

See merge request ruben/jail2ban-pf!1
2022-03-14 15:41:45 +00:00
4ca4233892 Feature/01 test all the things 2022-03-14 15:41:45 +00:00
13 changed files with 483 additions and 54 deletions

3
.gitignore vendored
View File

@ -12,3 +12,6 @@ htmlcov/
dist/
build/
*.egg-info/
coverage.xml
report.xml

View File

@ -1,15 +1,19 @@
run tests:
stage: test
image: python:3.8
image: python:3.9
script:
- pip install pytest pytest-cov pytest-mock pytest-flask
- pip install Flask-HTTPAuth
- coverage run -m pytest
- coverage run -m pytest --junitxml=report.xml
- coverage report
- coverage xml
coverage: '/^TOTAL.+?(\d+\%)$/'
artifacts:
when: always
reports:
cobertura: coverage.xml
coverage_report:
coverage_format: cobertura
path: coverage.xml
junit: report.xml
tags:
- docker

3
CHANGELOG Normal file
View File

@ -0,0 +1,3 @@
- 2023.1
* Implement #3, a /ping health check endpoint

130
README.md Normal file
View File

@ -0,0 +1,130 @@
[![pipeline status](https://gitlab.niet.verweg.com/ruben/jail2ban-pf/badges/main/pipeline.svg)](https://gitlab.niet.verweg.com/ruben/jail2ban-pf/-/commits/main)
[![coverage report](https://gitlab.niet.verweg.com/ruben/jail2ban-pf/badges/main/coverage.svg)](https://gitlab.niet.verweg.com/ruben/jail2ban-pf/-/commits/main)
## Installation
* Install uwsgi
```
sudo pkg install www/uwsgi
```
* Clone this repository
## Configuration
### rc.conf
* Use the following for configuring uwsgi in rc.conf
```
sudo sysrc uwsgi\_enable="YES"
sudo sysrc uwsgi\_profiles="jail2ban\_pf"
sudo sysrc uwsgi\_jail2ban\_pf\_flags="-L -M --uid \_jail2ban --python-path /opt/jail2ban-pf --wsgi-file /opt/jail2ban-pf/wsgi.py --stats 127.0.0.1:9191 --socket 127.0.0.1:3031 --chdir /var/empty --callable app --manage-script-name"
```
### jail2ban
* Configure <installation root>/instance/config.py
```
SECRET\_KEY = os.urandom(32).hex()
AUTHFILE = '/usr/local/etc/jail2ban-pf-users.txt'
```
### nginx
* Configure a nginx upstream and vhost
_Of course you can listen on ipv4/ipv6 but you want to protect these addresses from inadvertent or malicious probes_
upstream uwsgi_pf_jail2ban {
server 127.0.0.1:3031;
}
server {
listen unix:/path/to/jail_1/var/run/pf2ban/pf_jail2ban.sock;
listen unix:/path/to/jail_2/var/run/pf2ban/pf_jail2ban.sock;
listen unix:/path/to/jail_3/var/run/pf2ban/pf_jail2ban.sock;
server_name _;
location / {
index index.html index.htm index.php;
allow all;
include /usr/local/etc/nginx/uwsgi_params-dist;
uwsgi_pass uwsgi_pf_jail2ban;
}
}
### /etc/pf.conf
* Place anchors in pf for jail2ban to use. You probably want to place the early in your existing pf configuration
```
anchor "f2b/*"
anchor f2b-jail {
anchor "jail1_fqdn" to { <addr_jail1>, <addr_extra_jail1>, <addr_extra6_jail1> }
anchor "jail2_fqdn" to { <addr_jail2>, <addr_extra_jail2>, <addr_extra6_jail2> }
anchor "jail3_fqdn" to { <addr_jail3>, <addr_extra_jail3>, <addr_extra6_jail3> }
}
```
Having seperate anchors per jail makes it possible to have fine grained
blocking: Something that is harmful to jail2 might be perfectly legit for jail2.
#### Checking rules/tables made with fail2ban/jail2ban
Fail2ban will (re)create the per anchor rules on startup, and populate the designated address tables with offenders, e.g.:
sudo pfctl -a f2b-jail/jail1\_fqdn -T show -t f2b-recidive
192.0.2.66
2001:db8:abad:cafe:0bad:f00d
And the rules referencing these tables
sudo pfctl -a 'f2b-jail/jail1\_fqdn' -s rules
block drop quick proto tcp from <f2b-dovecot> to any port = pop3
block drop quick proto tcp from <f2b-dovecot> to any port = pop3s
block drop quick proto tcp from <f2b-dovecot> to any port = imap
block drop quick proto tcp from <f2b-dovecot> to any port = imaps
block drop quick proto tcp from <f2b-dovecot> to any port = submission
block drop quick proto tcp from <f2b-dovecot> to any port = smtps
block drop quick proto tcp from <f2b-dovecot> to any port = sieve
block drop quick proto tcp from <f2b-sendmail-auth> to any port = submission
block drop quick proto tcp from <f2b-sendmail-auth> to any port = smtps
block drop quick proto tcp from <f2b-sendmail-auth> to any port = smtp
block drop quick proto tcp from <f2b-sshd> to any port = ssh
block drop quick proto tcp from <f2b-recidive> to any
### fail2ban
* Create the following action plugin for fail2ban on the jail desiring to use fail2ban/jail2ban
```
cat <<'EOT' | tee /usr/local/etc/fail2ban/action.d/jail2ban-pf.conf > /dev/null
Definition]
actionstart = curl --unix-socket <jail2ban_sock> --basic -u '<jail2ban_user>:<jail2ban_pass>' -XPUT -H 'Content-Type: application/json' -d '{"port":"<actiontype>","name":"<name>","protocol":"<protocol>"}' http://localhost/register
actionstart_on_demand = false
actionstop = curl --unix-socket <jail2ban_sock> --basic -u '<jail2ban_user>:<jail2ban_pass>' -XDELETE -H 'Content-Type: application/json' -d '{"port":"<actiontype>","name":"<name>","protocol":"<protocol>"}' http://localhost/register
actionflush = curl --unix-socket <jail2ban_sock> --basic -u '<jail2ban_user>:<jail2ban_pass>' -X GET http://localhost/flush/<name>
actioncheck =
actionban = curl --unix-socket <jail2ban_sock> --basic -u '<jail2ban_user>:<jail2ban_pass>' -X PUT -H 'Content-Type: application/json' -d '{"name":"<name>","ip":"<ip>"}' http://localhost/ban
actionunban = curl --unix-socket <jail2ban_sock> --basic -u '<jail2ban_user>:<jail2ban_pass>' -X DELETE -H 'Content-Type: application/json' -d '{"name":"<name>","ip":"<ip>"}' http://localhost/ban
[Init]
protocol = tcp
actiontype = <multiport>
allports = any
multiport = any port {<port>}
jail2ban_sock = /var/run/pf2ban/jail2ban.sock
jail2ban_user = login as set in password file for jail2ban
jail2ban_pass = password as set in password file for jail2ban
```
* Configure jail.local
```
cat <<'EOT' | tee /usr/local/etc/fail2ban/jail.local > /dev/null
[DEFAULT]
banaction = jail2ban-pf
```

View File

@ -5,6 +5,7 @@ from ipaddress import ip_address
import re
from jail2ban.pfctl import pfctl_table_op, pfctl_cfg_read, pfctl_cfg_write
from jail2ban.auth import get_users
from subprocess import CalledProcessError
auth = HTTPBasicAuth()
@ -26,15 +27,11 @@ def untaint(pattern, string):
raise ValueError(f'"{string}" is tainted')
def create_app(config=None):
def create_app():
app = Flask(__name__, instance_relative_config=True)
if config is None:
# load the instance config, if it exists, when not testing
app.config.from_pyfile('config.py', silent=True)
else:
# load the test config if passed in
app.config.from_pyfile(config, silent=True)
@auth.verify_password
def verify_password(username, password):
@ -45,6 +42,15 @@ def create_app(config=None):
check_password_hash(users.get(username), password):
return username
@app.route("/ping", methods=['GET'])
@auth.login_required
def ping():
remote_user = auth.username()
app.logger.info('Received ping for'
f' anchor f2b-jail/{remote_user}')
return jsonify({'anchor': f'f2b-jail/{remote_user}',
'operation': 'ping',
'result': 'pong'})
@app.route("/flush/<name>", methods=['GET'])
@auth.login_required
def flush(name):
@ -58,7 +64,7 @@ def create_app(config=None):
return jsonify({'anchor': f'f2b-jail/{remote_user}',
'table': f'f2b-{name}',
'operation': 'flush',
'result': res})
'result': [x.decode('ascii') for x in res]})
@app.route("/register", methods=['PUT', 'DELETE'])
@auth.login_required
@ -80,7 +86,7 @@ def create_app(config=None):
])
res = pfctl_cfg_write(f'f2b-jail/{remote_user}',
b'\n'.join(cfg) + b'\n')
elif request.method == 'DELETE':
else: # 'DELETE':
cfg = [cfg_line for cfg_line in cfg
if cfg_line.find(bytes(f'<f2b-{name}>', 'ascii')) == -1]
res = pfctl_cfg_write(f'f2b-jail/{remote_user}',
@ -92,12 +98,11 @@ def create_app(config=None):
table=f'f2b-{name}',
operation='kill')
app.logger.info(f'pfctl -a f2b-jail/{remote_user} -f-')
return jsonify({'remote_user': remote_user, 'data': data})
return jsonify({'anchor': f'f2b-jail/{remote_user}',
'table': f'f2b-{name}',
'action': 'start' if request.method == 'PUT'
else 'stop',
'result': res})
'result': [x.decode('ascii') for x in res]})
@app.route("/ban", methods=['PUT', 'DELETE'])
@auth.login_required
@ -114,7 +119,7 @@ def create_app(config=None):
table=f'f2b-{name}',
operation='add',
value=str(ip))
elif request.method == 'DELETE':
else: # 'DELETE':
app.logger.info(f'Remove {ip} from f2b-{name}'
f' in anchor f2b-jail/{remote_user}')
res = pfctl_table_op(f'f2b-jail/{remote_user}',
@ -135,6 +140,22 @@ def create_app(config=None):
app.logger.fatal(error)
return jsonify({'error': str(error)}), 500
@app.errorhandler(CalledProcessError)
def subprocess_err(error):
'''
Show a json parsable error if the value is illegal
'''
app.logger.fatal(error)
return jsonify({'error': str(error)}), 500
@app.errorhandler(FileNotFoundError)
def filenotfound_err(error):
'''
Show a json parsable error if the value is illegal
'''
app.logger.fatal(error)
return jsonify({'error': str(error)}), 500
@auth.error_handler
def auth_error():
app.logger.error('Access Denied')

View File

@ -1,18 +1,15 @@
from flask import current_app, g
import os
from flask import current_app
def get_users():
if 'users' not in g:
users = {}
authfile = current_app.config['AUTHFILE']
current_app.logger.debug('Reading %s for users', authfile)
with current_app.open_resource(os.path.join("..",
authfile)) as f:
with current_app.open_resource(authfile) as f:
for entry in f:
users.update({tuple(entry.decode('ascii').strip().split(':', 1))})
g.users = users
current_app.logger.debug(g.users)
return g.users
users.update({
tuple(entry.decode('ascii').strip().split(':', 1))})
current_app.logger.debug(users)
return users

View File

@ -9,11 +9,9 @@ def pfctl_cfg_read(anchor):
cmd = [_SUDO, _PFCTL, '-a', anchor, '-sr']
logging.info('Running %s', cmd)
res = run(cmd, capture_output=True)
res = run(cmd, capture_output=True, check=True)
if res and res.stdout:
logging.info('Result: %s', res)
res.check_returncode()
return res.stdout.splitlines()
@ -24,12 +22,11 @@ def pfctl_cfg_write(anchor, cfg):
res = run(cmd,
input=cfg,
check=True,
capture_output=True)
if res:
logging.info('Result: %s', res)
res.check_returncode()
return res
return res.stdout.splitlines()
def pfctl_table_op(anchor, **kwargs):
@ -40,9 +37,9 @@ def pfctl_table_op(anchor, **kwargs):
logging.info('Running %s', cmd)
res = run([x for x in cmd if x is not None], capture_output=True)
res = run([x for x in cmd if x is not None],
capture_output=True,
check=True)
if res:
logging.debug(res)
res.check_returncode()
return res.stdout.splitlines()

View File

@ -1,4 +1,5 @@
import pytest
import base64
from jail2ban import create_app
@ -8,7 +9,7 @@ def app():
app.config.update({
"TESTING": True,
"SECRET_KEY": 'Testing',
"AUTHFILE": 'tests/users-test.txt'
"AUTHFILE": '../tests/users-test.txt'
})
# other setup can go here
@ -26,3 +27,8 @@ def client(app):
@pytest.fixture()
def runner(app):
return app.test_cli_runner()
@pytest.fixture()
def valid_credentials():
return base64.b64encode(b"test.example.com:testpassword").decode("utf-8")

102
tests/test_ban.py Normal file
View File

@ -0,0 +1,102 @@
from types import SimpleNamespace
def test_ban_ipv6(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res.stdout = b''
run_res.stderr = b'1/1 addresses added.\n'
run_res.returncode = 0
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"name": "sshd", "ip": "2001:db8::abad:cafe"}
response = client.put("/ban",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['operation'] == 'add'
def test_ban_ipv4(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res.stdout = b''
run_res.stderr = b'1/1 addresses added.\n'
run_res.returncode = 0
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"name": "sshd", "ip": "192.0.2.42"}
response = client.put("/ban",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['operation'] == 'add'
def test_ban_invalid(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res.stdout = b''
run_res.stderr = b'1/1 addresses added.\n'
run_res.returncode = 0
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"name": "sshd", "ip": "not:an::addr:ess"}
response = client.put("/ban",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['error'] == "'not:an::addr:ess' does not " \
"appear to be an IPv4 or IPv6 address"
def test_unban_ipv6(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res.stdout = b''
run_res.stderr = b'1/1 addresses deleted.\n'
run_res.returncode = 0
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"name": "sshd", "ip": "2001:db8::abad:cafe"}
response = client.delete("/ban",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['operation'] == 'delete'
def test_unban_ipv4(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res.stdout = b''
run_res.stderr = b'1/1 addresses deleted.\n'
run_res.returncode = 0
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"name": "sshd", "ip": "192.0.2.42"}
response = client.delete("/ban",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['operation'] == 'delete'

77
tests/test_flush.py Normal file
View File

@ -0,0 +1,77 @@
from types import SimpleNamespace
from subprocess import CalledProcessError
def test_flush(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res.stdout = b''
run_res.stderr = b'1/1 addresses deleted.\n'
run_res.returncode = 0
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
name = 'sshd'
response = client.get(f"/flush/{name}",
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['operation'] == 'flush'
def test_flush_nonexistent(client, mocker, valid_credentials):
cmd = ['/usr/local/bin/sudo',
'/sbin/pfctl', '-a', 'some/anchor',
'-t', 'nonexistent', '-T', 'flush']
side_effect = CalledProcessError(255, cmd, output=b'',
stderr=b'pfctl: Table does not exist')
mocker.patch('jail2ban.pfctl.run',
side_effect=side_effect)
name = 'nonexistent'
response = client.get(f"/flush/{name}",
headers={"Authorization":
"Basic " + valid_credentials})
assert 'error' in response.json
def test_wrong_method(client, mocker, valid_credentials):
cmd = ['/usr/local/bin/sudo',
'/sbin/pfctl', '-a', 'some/anchor',
'-t', 'nonexistent', '-T', 'flush']
side_effect = CalledProcessError(255, cmd, output=b'',
stderr=b'pfctl: Table does not exist')
mocker.patch('jail2ban.pfctl.run',
side_effect=side_effect)
name = 'nonexistent'
response = client.put(f"/flush/{name}",
headers={"Authorization":
"Basic " + valid_credentials})
assert response.status_code == 405
def test_filenotfound(app, mocker, valid_credentials):
app.config.update({
"AUTHFILE": '../tests/nonexistent-users-test.txt'
})
client = app.test_client()
name = 'nonexistent'
response = client.get(f"/flush/{name}",
headers={"Authorization":
"Basic " + valid_credentials})
assert response.status_code == 500

10
tests/test_ping.py Normal file
View File

@ -0,0 +1,10 @@
def test_ping(client, mocker, valid_credentials):
'''
Test application health check
'''
response = client.get("/ping",
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['operation'] == 'ping'

View File

@ -1,5 +1,4 @@
import base64
from types import SimpleNamespace
from subprocess import CompletedProcess
pfctl_stdout_lines = b'''
block drop quick proto tcp from <f2b-sendmail-auth> to any port = submission
@ -7,31 +6,108 @@ block drop quick proto tcp from <f2b-sendmail-auth> to any port = smtps
block drop quick proto tcp from <f2b-sendmail-auth> to any port = smtp
block drop quick proto tcp from <f2b-sshd> to any port = ssh
block drop quick proto tcp from <f2b-recidive> to any
'''
'''.strip() + b'\n'
pfctl_stdout_lines_scratch = b'table <f2b-dovecot> persist counters\n' \
b'block quick proto tcp from <f2b-dovecot>' \
b' to any port ' \
b'{pop3,pop3s,imap,imaps,submission,465,sieve}\n'
def test_request_unauth(client):
json_payload = {"port": "any port {pop3,pop3s,imap,imaps,submission,465,sieve}", "name": "dovecot", "protocol": "tcp"}
def test_register_unauth(client):
json_payload = {"port":
"any port {pop3,pop3s,imap,imaps,submission,465,sieve}",
"name": "dovecot", "protocol": "tcp"}
response = client.put("/register", json=json_payload)
assert response.json['error'] == 'Access Denied'
def test_request_example(client, mocker):
def test_unregister_valid(client, mocker, valid_credentials):
def noop():
pass
run_res = SimpleNamespace()
run_res = CompletedProcess(args=['true'], returncode=0)
run_res.stdout = pfctl_stdout_lines
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"port":
"any port {pop3,pop3s,imap,imaps,submission,465,sieve}",
"name": "dovecot", "protocol": "tcp"}
response = client.delete("/register",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['action'] == 'stop'
def test_register_valid(client, mocker, valid_credentials):
def noop():
pass
run_res = CompletedProcess(args=['true'], returncode=0)
run_res.stdout = pfctl_stdout_lines
run_res.check_returncode = noop
pfctl_run = mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"port":
"any port {pop3,pop3s,imap,imaps,submission,465,sieve}",
"name": "dovecot", "protocol": "tcp"}
valid_credentials = base64.b64encode(b"test.example.com:testpassword").decode("utf-8")
json_payload = {"port": "any port {pop3,pop3s,imap,imaps,submission,465,sieve}", "name": "dovecot", "protocol": "tcp"}
response = client.put("/register",
json=json_payload,
headers={"Authorization": "Basic " + valid_credentials})
headers={"Authorization":
"Basic " + valid_credentials})
pfctl_run_input_arg = pfctl_run.call_args_list[1][1]['input']
for existing_line in pfctl_stdout_lines.splitlines():
assert existing_line in pfctl_run_input_arg.splitlines()
assert response.json['action'] == 'start'
assert response.json['remote_user'] == 'test.example.com'
def test_register_valid_from_scratch(client, mocker, valid_credentials):
def noop():
pass
run_res = CompletedProcess(args=['true'], returncode=0)
run_res.stdout = b''
run_res.check_returncode = noop
pfctl_run = mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"port":
"any port {pop3,pop3s,imap,imaps,submission,465,sieve}",
"name": "dovecot", "protocol": "tcp"}
response = client.put("/register",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
pfctl_run_input_arg = pfctl_run.call_args_list[1][1]['input']
assert pfctl_run_input_arg == pfctl_stdout_lines_scratch
assert response.json['action'] == 'start'
def test_register_invalid(client, mocker, valid_credentials):
def noop():
pass
run_res = CompletedProcess(args=['true'], returncode=0)
run_res.stdout = pfctl_stdout_lines
run_res.check_returncode = noop
mocker.patch('jail2ban.pfctl.run', return_value=run_res)
json_payload = {"port":
"not a pf statement",
"name": "dovecot", "protocol": "tcp"}
response = client.put("/register",
json=json_payload,
headers={"Authorization":
"Basic " + valid_credentials})
assert response.json['error'] == '"not a pf statement" is tainted'

3
wsgi.py Normal file
View File

@ -0,0 +1,3 @@
from jail2ban import create_app
app = create_app()