Can the same algorithm (like RSA) be used for both encryption and digital signatures? How?

Yes — RSA does both; for signing, the encrypt/decrypt operations are essentially "swapped" (you sign with the private key, verify with the public key).

In asymmetric encryption: encrypt with the public key, decrypt with the private key. In a signature: the operations are reversed — the signer uses their private key to produce the signature, and anyone verifies with the public key.

That's why a signature proves authorship: only the private-key holder could have created it, yet everyone can check it.

Tip: Encrypt-for-secrecy uses the recipient's keys (public→private). Sign-for-proof uses the sender's keys (private→public). Same math, opposite direction.