Install the target with rustup target add aarch64-unknown-linux-gnu, add the gcc cross linker from your distro, and point cargo at it in .cargo/config.toml. That is genuinely all it takes for most crates.
Crates with C dependencies need a sysroot; for those, cross-rs is still the easier path.