[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:29 dist-test-slave-9tkh krb5kdc[22584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:29 dist-test-slave-9tkh krb5kdc[22584](info): set up 2 sockets
May 01 02:07:29 dist-test-slave-9tkh krb5kdc[22584](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240501 02:07:30.310209 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:30.426560 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45731
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:45731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:30.505306 22597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:30.505515 22597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:30.505546 22597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:30.506887 22597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:30.506938 22597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:30.506958 22597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:30.506973 22597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:30.506985 22597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:30.508610 22597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:45731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:30.508826 22597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:30.509009 22597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:30.513626 22602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:30.513975 22603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:30.514554 22605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:30.514977 22597 server_base.cc:1034] running on GCE node
I20240501 02:07:30.515197 22597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:30.515415 22597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:30.516551 22597 hybrid_clock.cc:648] HybridClock initialized: now 1714529250516539 us; error 76 us; skew 500 ppm
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:30.523226 22597 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:30.523671 22597 webserver.cc:460] Webserver started at http://127.22.12.126:39005/ using document root <none> and password file <none>
I20240501 02:07:30.523875 22597 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:30.523917 22597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:30.524031 22597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:30.525024 22597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "29ff8f5f3b6847d4814f846001916753"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "ee9e92f521e4898bd01f70e961005005"
server_key_iv: "4c3b46ee45d471ad73e28c2466a29e1b"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.525336 22597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "29ff8f5f3b6847d4814f846001916753"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "ee9e92f521e4898bd01f70e961005005"
server_key_iv: "4c3b46ee45d471ad73e28c2466a29e1b"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.526451 22597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 02:07:30.527112 22612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.527264 22597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.527348 22597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "29ff8f5f3b6847d4814f846001916753"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "ee9e92f521e4898bd01f70e961005005"
server_key_iv: "4c3b46ee45d471ad73e28c2466a29e1b"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.527405 22597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:30.539456 22597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:30.541234 22597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:30.541354 22597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:30.545115 22597 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:45731
I20240501 02:07:30.545150 22664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:45731 every 8 connection(s)
I20240501 02:07:30.545522 22597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:30.546130 22665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:30.548334 22665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753: Bootstrap starting.
I20240501 02:07:30.548903 22665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:30.549152 22665 log.cc:824] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:30.549856 22665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753: No bootstrap required, opened a new log
I20240501 02:07:30.551065 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 22597
I20240501 02:07:30.551151 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:30.551378 22665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8f5f3b6847d4814f846001916753" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45731 } }
I20240501 02:07:30.551510 22665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:30.551534 22665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ff8f5f3b6847d4814f846001916753, State: Initialized, Role: FOLLOWER
I20240501 02:07:30.551767 22577 external_mini_cluster.cc:1428] Setting key c4b4b8df0bcea3a1fa355ac34b2a7a2f
I20240501 02:07:30.551774 22665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8f5f3b6847d4814f846001916753" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45731 } }
I20240501 02:07:30.551852 22665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:30.551880 22665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:30.551918 22665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:30.552593 22665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8f5f3b6847d4814f846001916753" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45731 } }
I20240501 02:07:30.552770 22665 leader_election.cc:302] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ff8f5f3b6847d4814f846001916753; no voters:
I20240501 02:07:30.552927 22665 leader_election.cc:288] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:30.553165 22665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:30.553179 22670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:30.553381 22670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [term 1 LEADER]: Becoming Leader. State: Replica: 29ff8f5f3b6847d4814f846001916753, State: Running, Role: LEADER
I20240501 02:07:30.553526 22670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8f5f3b6847d4814f846001916753" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45731 } }
I20240501 02:07:30.553912 22671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ff8f5f3b6847d4814f846001916753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8f5f3b6847d4814f846001916753" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45731 } } }
I20240501 02:07:30.553947 22672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ff8f5f3b6847d4814f846001916753. Latest consensus state: current_term: 1 leader_uuid: "29ff8f5f3b6847d4814f846001916753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ff8f5f3b6847d4814f846001916753" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45731 } } }
I20240501 02:07:30.554008 22671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:30.554015 22672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:30.554206 22679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:30.554782 22679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:30.556268 22679 catalog_manager.cc:1347] Generated new cluster ID: c1925b04f2694448a7a637a1b1dcbd67
I20240501 02:07:30.556329 22679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:30.564090 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:30.552499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.22.12.126:45731)
0501 02:07:30.552680 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:30.552693 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:30.552868 (+ 175us) server_negotiation.cc:407] Connection header received
0501 02:07:30.554779 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:30.554787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:30.554841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.554962 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:30.556792 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.557523 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.558130 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.558376 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.561028 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.561035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:30.561035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:30.561056 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:30.562634 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.562984 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.562987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:30.562988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:30.563035 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.563205 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.563206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:30.563206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:30.563407 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.563538 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:30.563760 (+ 222us) server_negotiation.cc:299] Negotiation successful
0501 02:07:30.563876 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20240501 02:07:30.568472 22679 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:30.569180 22679 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:30.575654 22679 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 29ff8f5f3b6847d4814f846001916753: Generated new TSK 0
I20240501 02:07:30.575845 22679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:30.606714 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45731
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:30.706897 22691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:30.707091 22691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:30.707131 22691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:30.708485 22691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:30.708534 22691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:30.708568 22691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:30.710148 22691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:30.710374 22691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:30.710615 22691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:30.715590 22699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:30.715966 22697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:30.716446 22696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:30.716928 22691 server_base.cc:1034] running on GCE node
I20240501 02:07:30.717111 22691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:30.717330 22691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:30.718479 22691 hybrid_clock.cc:648] HybridClock initialized: now 1714529250718459 us; error 40 us; skew 500 ppm
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:30.721197 22691 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:30.721808 22691 webserver.cc:460] Webserver started at http://127.22.12.65:34437/ using document root <none> and password file <none>
I20240501 02:07:30.722055 22691 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:30.722111 22691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:30.722222 22691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:30.723572 22691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "594c249181d942629279d3d3a25cf30c"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "34269834552a6ca2374347adc869bad0"
server_key_iv: "8b3c56fc76d8f38727f34982caf9a6a8"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.723966 22691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "594c249181d942629279d3d3a25cf30c"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "34269834552a6ca2374347adc869bad0"
server_key_iv: "8b3c56fc76d8f38727f34982caf9a6a8"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.725579 22691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:07:30.726393 22706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.726593 22691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:30.726684 22691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "594c249181d942629279d3d3a25cf30c"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "34269834552a6ca2374347adc869bad0"
server_key_iv: "8b3c56fc76d8f38727f34982caf9a6a8"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.726749 22691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:30.751066 22691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:30.753268 22691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:30.753373 22691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:30.753573 22691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:30.753867 22691 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:30.753899 22691 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.753929 22691 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:30.753944 22691 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.759558 22691 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:40063
I20240501 02:07:30.759615 22819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:40063 every 8 connection(s)
I20240501 02:07:30.759884 22691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:30.761838 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 22691
I20240501 02:07:30.761935 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:30.762076 22577 external_mini_cluster.cc:1428] Setting key 1e0cb21e7f0046881d696d87e24390fa
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:30.768877 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:30.760541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:41287 (local address 127.22.12.126:45731)
0501 02:07:30.760596 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:30.760598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:30.760763 (+ 165us) server_negotiation.cc:407] Connection header received
0501 02:07:30.761233 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:30.761233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:30.761245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.761305 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:30.762380 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.763069 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.763601 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.763752 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.765856 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.765861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:30.765861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:30.765867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:30.767273 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.767712 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.767712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:30.767713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:30.767729 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.768040 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.768041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:30.768042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:30.768170 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.768247 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:30.768652 (+ 405us) server_negotiation.cc:299] Negotiation successful
0501 02:07:30.768772 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:07:30.768960 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:30.760519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45731 (local address 127.22.12.65:41287)
0501 02:07:30.760653 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:30.760683 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:30.761143 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:30.761348 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:30.761352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:30.761636 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:30.762289 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:30.762296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.763157 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.763158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:30.763525 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:30.763527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.763630 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.764114 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:30.764139 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:30.765734 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:30.767356 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.767363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:30.767366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:30.767637 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.767784 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.767784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:30.767785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:30.767889 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.768259 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:30.768266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:30.768440 (+ 174us) client_negotiation.cc:769] Sending connection context
0501 02:07:30.768596 (+ 156us) client_negotiation.cc:240] Negotiation successful
0501 02:07:30.768743 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20240501 02:07:30.769469 22820 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45731
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
I20240501 02:07:30.769578 22820 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:30.769809 22820 heartbeater.cc:511] Master 127.22.12.126:45731 requested a full tablet report, sending...
I20240501 02:07:30.770291 22629 ts_manager.cc:194] Registered new tserver with Master: 594c249181d942629279d3d3a25cf30c (127.22.12.65:40063)
I20240501 02:07:30.770959 22629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:41287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:30.789557 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45731
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:30.867892 22825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:30.868078 22825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:30.868099 22825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:30.869415 22825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:30.869480 22825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:30.869508 22825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:30.871091 22825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:30.871301 22825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:30.871488 22825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:30.875983 22830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:30.876179 22831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:30.876586 22833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:30.877121 22825 server_base.cc:1034] running on GCE node
I20240501 02:07:30.877288 22825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:30.877488 22825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:30.878573 22825 hybrid_clock.cc:648] HybridClock initialized: now 1714529250878563 us; error 32 us; skew 500 ppm
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:30.880932 22825 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:30.881350 22825 webserver.cc:460] Webserver started at http://127.22.12.66:38499/ using document root <none> and password file <none>
I20240501 02:07:30.881527 22825 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:30.881569 22825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:30.881675 22825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:30.882699 22825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "ff668e47c5e2418dabbecf0fe71e88ff"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "d933c3ffbec8bf8bac83b2d7d5527dd1"
server_key_iv: "572e06ecb7dfc71635b12f27d6166d8b"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.883003 22825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "ff668e47c5e2418dabbecf0fe71e88ff"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "d933c3ffbec8bf8bac83b2d7d5527dd1"
server_key_iv: "572e06ecb7dfc71635b12f27d6166d8b"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.884207 22825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:30.884827 22840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.884982 22825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:30.885054 22825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "ff668e47c5e2418dabbecf0fe71e88ff"
format_stamp: "Formatted at 2024-05-01 02:07:30 on dist-test-slave-9tkh"
server_key: "d933c3ffbec8bf8bac83b2d7d5527dd1"
server_key_iv: "572e06ecb7dfc71635b12f27d6166d8b"
server_key_version: "encryptionkey@0"
I20240501 02:07:30.885111 22825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:30.899725 22825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:30.901434 22825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:30.901552 22825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:30.901760 22825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:30.902077 22825 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:30.902110 22825 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.902139 22825 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:30.902155 22825 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:30.907560 22825 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:40645
I20240501 02:07:30.907613 22953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:40645 every 8 connection(s)
I20240501 02:07:30.907898 22825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:07:30 dist-test-slave-9tkh krb5kdc[22584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529250, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:30.914016 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 22825
I20240501 02:07:30.914111 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:30.914247 22577 external_mini_cluster.cc:1428] Setting key f319e9d594e295a186a998fdff7857fb
I20240501 02:07:30.915648 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:30.908534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:34939 (local address 127.22.12.126:45731)
0501 02:07:30.908578 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:30.908580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:30.908754 (+ 174us) server_negotiation.cc:407] Connection header received
0501 02:07:30.909215 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:30.909216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:30.909225 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.909283 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:30.910231 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.910751 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.911268 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.911371 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.913149 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.913154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:30.913154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:30.913161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:30.914420 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.914834 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.914834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:30.914835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:30.914860 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.915101 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.915102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:30.915102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:30.915184 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.915230 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:30.915511 (+ 281us) server_negotiation.cc:299] Negotiation successful
0501 02:07:30.915582 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:07:30.915807 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:30.908504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45731 (local address 127.22.12.66:34939)
0501 02:07:30.908658 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:30.908688 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:30.909130 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:30.909340 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:30.909343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:30.909632 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:30.910151 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:30.910155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.910826 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.910827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:30.911207 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:30.911208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.911314 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.911760 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:30.911773 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:30.913034 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:30.914521 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.914549 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:30.914552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:30.914768 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.914946 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.914947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:30.914948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:30.915024 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.915235 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:30.915239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:30.915383 (+ 144us) client_negotiation.cc:769] Sending connection context
0501 02:07:30.915500 (+ 117us) client_negotiation.cc:240] Negotiation successful
0501 02:07:30.915628 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20240501 02:07:30.916224 22954 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45731
I20240501 02:07:30.916319 22954 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:30.916555 22954 heartbeater.cc:511] Master 127.22.12.126:45731 requested a full tablet report, sending...
I20240501 02:07:30.917037 22629 ts_manager.cc:194] Registered new tserver with Master: ff668e47c5e2418dabbecf0fe71e88ff (127.22.12.66:40645)
I20240501 02:07:30.917500 22629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:34939
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:30.938129 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45731
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:31.016489 22959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:31.016680 22959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:31.016700 22959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:31.017982 22959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:31.018020 22959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:31.018043 22959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:31.019623 22959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:31.019812 22959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:31.019968 22959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:31.024369 22965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:31.024645 22964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:31.025054 22967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:31.025420 22959 server_base.cc:1034] running on GCE node
I20240501 02:07:31.025585 22959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:31.025787 22959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:31.026916 22959 hybrid_clock.cc:648] HybridClock initialized: now 1714529251026900 us; error 28 us; skew 500 ppm
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529251, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:31.029269 22959 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:31.029688 22959 webserver.cc:460] Webserver started at http://127.22.12.67:43199/ using document root <none> and password file <none>
I20240501 02:07:31.029896 22959 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:31.029939 22959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:31.030078 22959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:31.031222 22959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "5f9421515d0546cd89598c235b0d415a"
format_stamp: "Formatted at 2024-05-01 02:07:31 on dist-test-slave-9tkh"
server_key: "49ffc88275172bbdec4c9db089b0e928"
server_key_iv: "96ca81ee78483ee33019a59eb57a705d"
server_key_version: "encryptionkey@0"
I20240501 02:07:31.031562 22959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "5f9421515d0546cd89598c235b0d415a"
format_stamp: "Formatted at 2024-05-01 02:07:31 on dist-test-slave-9tkh"
server_key: "49ffc88275172bbdec4c9db089b0e928"
server_key_iv: "96ca81ee78483ee33019a59eb57a705d"
server_key_version: "encryptionkey@0"
I20240501 02:07:31.032761 22959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:31.033387 22974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:31.033537 22959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:31.033612 22959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "5f9421515d0546cd89598c235b0d415a"
format_stamp: "Formatted at 2024-05-01 02:07:31 on dist-test-slave-9tkh"
server_key: "49ffc88275172bbdec4c9db089b0e928"
server_key_iv: "96ca81ee78483ee33019a59eb57a705d"
server_key_version: "encryptionkey@0"
I20240501 02:07:31.033669 22959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:31.051399 22959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:31.053321 22959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:31.053432 22959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:31.053681 22959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:31.054036 22959 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:31.054069 22959 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:31.054098 22959 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:31.054116 22959 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:31.059487 22959 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:34957
I20240501 02:07:31.059533 23087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:34957 every 8 connection(s)
I20240501 02:07:31.059809 22959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:31.062485 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 22959
I20240501 02:07:31.062593 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:31.062748 22577 external_mini_cluster.cc:1428] Setting key 63d5e2a85f3d0197c666b79aa39ac302
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[22584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529251, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:31.067811 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:31.060404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:59707 (local address 127.22.12.126:45731)
0501 02:07:31.060445 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:31.060448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:31.060635 (+ 187us) server_negotiation.cc:407] Connection header received
0501 02:07:31.061109 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:31.061111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:31.061124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.061180 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:31.062153 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.062861 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.063551 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.063718 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.065598 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:31.065602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:31.065602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:31.065608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:31.066819 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.067188 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.067189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:31.067190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:31.067211 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.067356 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.067356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:31.067356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:31.067434 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:31.067477 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:31.067681 (+ 204us) server_negotiation.cc:299] Negotiation successful
0501 02:07:31.067745 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:07:31.067912 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:31.060391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45731 (local address 127.22.12.67:59707)
0501 02:07:31.060525 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:31.060550 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:31.061010 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:31.061241 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:31.061245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:31.061543 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:31.062050 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:31.062054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.062962 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.062963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:31.063487 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:31.063490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.063591 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.064113 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:31.064137 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:31.065506 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:31.066906 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:31.066911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:31.066914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:31.067119 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:31.067267 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:31.067267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:31.067268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:31.067316 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:31.067473 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:31.067476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:31.067581 (+ 105us) client_negotiation.cc:769] Sending connection context
0501 02:07:31.067662 (+ 81us) client_negotiation.cc:240] Negotiation successful
0501 02:07:31.067762 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20240501 02:07:31.068396 23088 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45731
I20240501 02:07:31.068478 23088 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:31.068667 23088 heartbeater.cc:511] Master 127.22.12.126:45731 requested a full tablet report, sending...
I20240501 02:07:31.069036 22629 ts_manager.cc:194] Registered new tserver with Master: 5f9421515d0546cd89598c235b0d415a (127.22.12.67:34957)
I20240501 02:07:31.069394 22629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:59707
I20240501 02:07:31.074760 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529251, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[22584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529251, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:31.095496 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:31.088872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.22.12.65:40063)
0501 02:07:31.089065 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:31.089072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:31.089082 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:31.089109 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:31.089111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:31.089123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.089189 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:31.089825 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.090570 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.091131 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.091292 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.093134 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:31.093145 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:31.093146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:31.093152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:31.094729 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.094978 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.094980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:31.094981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:31.095006 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.095099 (+ 93us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.095100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:31.095100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:31.095188 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:31.095295 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:31.095329 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:07:31.095394 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
W20240501 02:07:31.095849 22734 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41022
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[22584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:31.115796 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:31.109621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.22.12.65:40063)
0501 02:07:31.109682 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:31.109684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:31.109736 (+ 52us) server_negotiation.cc:407] Connection header received
0501 02:07:31.109810 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:31.109810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:31.109824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.109870 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:31.110501 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.111106 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.111678 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.111849 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.113705 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:31.113709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:31.113710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:31.113716 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:31.114930 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.115203 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.115204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:31.115204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:31.115225 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.115352 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.115353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:31.115353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:31.115427 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:31.115508 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:31.115620 (+ 112us) server_negotiation.cc:299] Negotiation successful
0501 02:07:31.115702 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:07:31.116282 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 22691
I20240501 02:07:31.120743 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 22825
I20240501 02:07:31.125512 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 22959
I20240501 02:07:31.129371 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 22597
2024-05-01T02:07:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1723 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[23102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[23102](info): set up 2 sockets
May 01 02:07:31 dist-test-slave-9tkh krb5kdc[23102](info): commencing operation
krb5kdc: starting...
W20240501 02:07:32.026185 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:32.142248 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:33967
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:33967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:32.219657 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:32.219856 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:32.219885 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:32.221246 23115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:32.221293 23115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:32.221311 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:32.221326 23115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:32.221339 23115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:32.222982 23115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:33967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:33967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:32.223199 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:32.223377 23115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:32.227555 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.227924 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.228425 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:32.228782 23115 server_base.cc:1034] running on GCE node
I20240501 02:07:32.229025 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:32.229243 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:32.230382 23115 hybrid_clock.cc:648] HybridClock initialized: now 1714529252230365 us; error 34 us; skew 500 ppm
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:32.233024 23115 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:32.233426 23115 webserver.cc:460] Webserver started at http://127.22.12.126:39055/ using document root <none> and password file <none>
I20240501 02:07:32.233575 23115 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:32.233609 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:32.233690 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:32.234714 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "1b251d704ce340a98cb0fcd707154975"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "275bc1888fc4386b6ad2cb310444e5b3"
server_key_iv: "c0cd158acffee55d38e0104d5930d740"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.234982 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "1b251d704ce340a98cb0fcd707154975"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "275bc1888fc4386b6ad2cb310444e5b3"
server_key_iv: "c0cd158acffee55d38e0104d5930d740"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.236158 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:32.236768 23130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.236919 23115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.236990 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "1b251d704ce340a98cb0fcd707154975"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "275bc1888fc4386b6ad2cb310444e5b3"
server_key_iv: "c0cd158acffee55d38e0104d5930d740"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.237053 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:32.246714 23115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:32.248628 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:32.248732 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:32.252298 23115 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:33967
I20240501 02:07:32.252328 23182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:33967 every 8 connection(s)
I20240501 02:07:32.252632 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:32.253423 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:32.255570 23183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975: Bootstrap starting.
I20240501 02:07:32.256070 23183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:32.256271 23183 log.cc:824] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:32.256455 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23115
I20240501 02:07:32.256549 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:32.256709 22577 external_mini_cluster.cc:1428] Setting key 0d71eba2a5ee124140f8e11b2e6ecf99
I20240501 02:07:32.257063 23183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975: No bootstrap required, opened a new log
I20240501 02:07:32.259449 23183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b251d704ce340a98cb0fcd707154975" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33967 } }
I20240501 02:07:32.259593 23183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:32.259613 23183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b251d704ce340a98cb0fcd707154975, State: Initialized, Role: FOLLOWER
I20240501 02:07:32.259702 23183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b251d704ce340a98cb0fcd707154975" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33967 } }
I20240501 02:07:32.259761 23183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:32.259795 23183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:32.259835 23183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:32.260335 23183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b251d704ce340a98cb0fcd707154975" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33967 } }
I20240501 02:07:32.260437 23183 leader_election.cc:302] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b251d704ce340a98cb0fcd707154975; no voters:
I20240501 02:07:32.260919 23183 leader_election.cc:288] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:32.260954 23188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:32.261059 23188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [term 1 LEADER]: Becoming Leader. State: Replica: 1b251d704ce340a98cb0fcd707154975, State: Running, Role: LEADER
I20240501 02:07:32.261148 23188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b251d704ce340a98cb0fcd707154975" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33967 } }
I20240501 02:07:32.261175 23183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:32.261551 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b251d704ce340a98cb0fcd707154975. Latest consensus state: current_term: 1 leader_uuid: "1b251d704ce340a98cb0fcd707154975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b251d704ce340a98cb0fcd707154975" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33967 } } }
I20240501 02:07:32.261551 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b251d704ce340a98cb0fcd707154975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b251d704ce340a98cb0fcd707154975" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33967 } } }
I20240501 02:07:32.261654 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:32.261669 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975 [sys.catalog]: This master's current role is: LEADER
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:32.262593 23192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:32.263051 23192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:32.264607 23192 catalog_manager.cc:1347] Generated new cluster ID: 53b75833f32d4915a685cca8b0774065
I20240501 02:07:32.264676 23192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:32.266096 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.257184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.22.12.126:33967)
0501 02:07:32.257318 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.257331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.257356 (+ 25us) server_negotiation.cc:407] Connection header received
0501 02:07:32.257823 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.257831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.257886 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.258033 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.258694 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.259866 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.260395 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.260621 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.262979 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.262987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.262988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.263007 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.264933 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.265220 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.265222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.265222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.265266 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.265401 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.265401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.265402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.265579 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.265715 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.265795 (+ 80us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.265912 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20240501 02:07:32.270653 23192 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:32.271173 23192 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:32.274096 23192 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1b251d704ce340a98cb0fcd707154975: Generated new TSK 0
I20240501 02:07:32.274271 23192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:32.302444 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33967
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:32.382383 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:32.382603 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:32.382635 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:32.384006 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:32.384052 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:32.384075 23209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:32.385682 23209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:32.385877 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:32.386060 23209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:32.390761 23215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.391220 23217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.391649 23214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:32.392143 23209 server_base.cc:1034] running on GCE node
I20240501 02:07:32.392342 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:32.392557 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:32.393694 23209 hybrid_clock.cc:648] HybridClock initialized: now 1714529252393670 us; error 42 us; skew 500 ppm
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:32.396256 23209 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:32.396663 23209 webserver.cc:460] Webserver started at http://127.22.12.65:35735/ using document root <none> and password file <none>
I20240501 02:07:32.396835 23209 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:32.396874 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:32.396981 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:32.397938 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "5092a49cccec4cb8b2bee72b9e12fd3c"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "f46996582aa166303b60e47553227639"
server_key_iv: "986436c6371eae580219ebefa4aa282c"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.398226 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "5092a49cccec4cb8b2bee72b9e12fd3c"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "f46996582aa166303b60e47553227639"
server_key_iv: "986436c6371eae580219ebefa4aa282c"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.399390 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:32.400027 23224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.400189 23209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:32.400262 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "5092a49cccec4cb8b2bee72b9e12fd3c"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "f46996582aa166303b60e47553227639"
server_key_iv: "986436c6371eae580219ebefa4aa282c"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.400318 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:32.408972 23209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:32.411023 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:32.411135 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:32.411317 23209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:32.411613 23209 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:32.411645 23209 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.411666 23209 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:32.411680 23209 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.417075 23209 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38545
I20240501 02:07:32.417117 23337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38545 every 8 connection(s)
I20240501 02:07:32.417402 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:32.425336 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.418032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:49541 (local address 127.22.12.126:33967)
0501 02:07:32.418107 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.418109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.418241 (+ 132us) server_negotiation.cc:407] Connection header received
0501 02:07:32.418728 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.418729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.418738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.418788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.419527 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.420021 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.420956 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.421078 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.422963 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.422967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.422968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.422974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.424246 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.424554 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.424555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.424555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.424579 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.424761 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.424763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.424763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.424882 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.424944 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.425187 (+ 243us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.425271 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 02:07:32.425379 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.418002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:33967 (local address 127.22.12.65:49541)
0501 02:07:32.418134 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.418161 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.418640 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.418846 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.418849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.419138 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:32.419423 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.419427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.420120 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.420121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.420878 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.420880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.420991 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.421482 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:32.421495 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:32.422859 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:32.424307 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:32.424310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:32.424312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:32.424497 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:32.424633 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:32.424633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:32.424634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:32.424680 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:32.424937 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:32.424939 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:32.425040 (+ 101us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.425135 (+ 95us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.425235 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20240501 02:07:32.425885 23338 heartbeater.cc:347] Connected to a master server at 127.22.12.126:33967
I20240501 02:07:32.425993 23338 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:32.426205 23338 heartbeater.cc:511] Master 127.22.12.126:33967 requested a full tablet report, sending...
I20240501 02:07:32.426728 23147 ts_manager.cc:194] Registered new tserver with Master: 5092a49cccec4cb8b2bee72b9e12fd3c (127.22.12.65:38545)
I20240501 02:07:32.426962 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23209
I20240501 02:07:32.427042 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:32.427258 22577 external_mini_cluster.cc:1428] Setting key de43bc72008b4c1a114ace5f79085c13
I20240501 02:07:32.427409 23147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:49541
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:32.453701 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33967
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:32.532163 23343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:32.532375 23343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:32.532407 23343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:32.533771 23343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:32.533821 23343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:32.533859 23343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:32.535555 23343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:32.535813 23343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:32.536001 23343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:32.540588 23348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.540751 23351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.541227 23349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:32.541642 23343 server_base.cc:1034] running on GCE node
I20240501 02:07:32.541817 23343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:32.542040 23343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:32.543164 23343 hybrid_clock.cc:648] HybridClock initialized: now 1714529252543156 us; error 30 us; skew 500 ppm
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:32.545614 23343 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:32.546051 23343 webserver.cc:460] Webserver started at http://127.22.12.66:44531/ using document root <none> and password file <none>
I20240501 02:07:32.546227 23343 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:32.546267 23343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:32.546370 23343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:32.547394 23343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "a6ec6b6cc7674123ab7feddbefd280dd"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "b0cf37b4b9bf2794407317b9deaf43b4"
server_key_iv: "fffa68fe5111666987707e7a84dc1af2"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.547677 23343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "a6ec6b6cc7674123ab7feddbefd280dd"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "b0cf37b4b9bf2794407317b9deaf43b4"
server_key_iv: "fffa68fe5111666987707e7a84dc1af2"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.548779 23343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:32.549448 23358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.549585 23343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:32.549652 23343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "a6ec6b6cc7674123ab7feddbefd280dd"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "b0cf37b4b9bf2794407317b9deaf43b4"
server_key_iv: "fffa68fe5111666987707e7a84dc1af2"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.549706 23343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:32.559116 23343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:32.560755 23343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:32.560863 23343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:32.561151 23343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:32.561468 23343 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:32.561501 23343 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.561530 23343 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:32.561574 23343 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.566751 23343 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:41707
I20240501 02:07:32.566792 23471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:41707 every 8 connection(s)
I20240501 02:07:32.567072 23343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:32.568048 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23343
I20240501 02:07:32.568152 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:32.568316 22577 external_mini_cluster.cc:1428] Setting key 9ae51d9e93950dbe6a593d93f485699e
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
I20240501 02:07:32.576058 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.567717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:38679 (local address 127.22.12.126:33967)
0501 02:07:32.567800 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.567802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.568037 (+ 235us) server_negotiation.cc:407] Connection header received
0501 02:07:32.568634 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.568635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.568645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.568700 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.569623 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.570098 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.571119 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.571291 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.573331 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.573337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.573337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.573345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.574874 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.575256 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.575258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.575258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.575286 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.575450 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.575451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.575451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.575571 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.575642 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.575857 (+ 215us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.575961 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 02:07:32.576110 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.567696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:33967 (local address 127.22.12.66:38679)
0501 02:07:32.567875 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.567910 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.568512 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.568750 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.568754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.569131 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:32.569529 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.569535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.570182 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.570183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.571032 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.571033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.571139 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.571628 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:32.571644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:32.573223 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:32.574961 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:32.574967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:32.574969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:32.575182 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:32.575350 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:32.575350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:32.575351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:32.575400 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:32.575628 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:32.575631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:32.575737 (+ 106us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.575819 (+ 82us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.575951 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20240501 02:07:32.576671 23472 heartbeater.cc:347] Connected to a master server at 127.22.12.126:33967
I20240501 02:07:32.576759 23472 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:32.576941 23472 heartbeater.cc:511] Master 127.22.12.126:33967 requested a full tablet report, sending...
I20240501 02:07:32.577296 23146 ts_manager.cc:194] Registered new tserver with Master: a6ec6b6cc7674123ab7feddbefd280dd (127.22.12.66:41707)
I20240501 02:07:32.577663 23146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:38679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:32.593887 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33967
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:32.674901 23477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:32.675096 23477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:32.675119 23477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:32.676493 23477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:32.676548 23477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:32.676576 23477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:32.678261 23477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:32.678483 23477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:32.678707 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:32.683367 23485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.683481 23483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:32.683864 23482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:32.684448 23477 server_base.cc:1034] running on GCE node
I20240501 02:07:32.684624 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:32.684819 23477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:32.685966 23477 hybrid_clock.cc:648] HybridClock initialized: now 1714529252685944 us; error 35 us; skew 500 ppm
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:32.688470 23477 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:32.688896 23477 webserver.cc:460] Webserver started at http://127.22.12.67:44637/ using document root <none> and password file <none>
I20240501 02:07:32.689095 23477 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:32.689136 23477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:32.689237 23477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:32.690209 23477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "925c8e44c1104f8d96088bd92398afc0"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "c4b18d551846a69bfc698c63bd3f9cb1"
server_key_iv: "19fa9b5405b6790d90b924d878a0a42d"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.690500 23477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "925c8e44c1104f8d96088bd92398afc0"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "c4b18d551846a69bfc698c63bd3f9cb1"
server_key_iv: "19fa9b5405b6790d90b924d878a0a42d"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.691656 23477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:32.692301 23492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.692456 23477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:32.692528 23477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "925c8e44c1104f8d96088bd92398afc0"
format_stamp: "Formatted at 2024-05-01 02:07:32 on dist-test-slave-9tkh"
server_key: "c4b18d551846a69bfc698c63bd3f9cb1"
server_key_iv: "19fa9b5405b6790d90b924d878a0a42d"
server_key_version: "encryptionkey@0"
I20240501 02:07:32.692589 23477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:32.701334 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:32.703435 23477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:32.703550 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:32.703730 23477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:32.704023 23477 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:32.704061 23477 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.704088 23477 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:32.704115 23477 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:32.709084 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:42509
I20240501 02:07:32.709138 23605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:42509 every 8 connection(s)
I20240501 02:07:32.709394 23477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:32.717610 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.710109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:52147 (local address 127.22.12.126:33967)
0501 02:07:32.710189 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.710192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.710294 (+ 102us) server_negotiation.cc:407] Connection header received
0501 02:07:32.710787 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.710788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.710798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.710856 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.711660 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.712165 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.713123 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.713260 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.715111 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.715115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.715115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.715122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.716426 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.716791 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.716792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.716793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.716811 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.717021 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.717022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.717022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.717125 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.717170 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.717459 (+ 289us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.717535 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 02:07:32.717736 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.710058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:33967 (local address 127.22.12.67:52147)
0501 02:07:32.710193 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.710219 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.710686 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.710912 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.710915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.711226 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:32.711553 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.711558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.712247 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.712248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.713013 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.713015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.713130 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.713622 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:32.713638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:32.714975 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:32.716515 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:32.716521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:32.716524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:32.716729 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:32.716888 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:32.716889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:32.716889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:32.716966 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:32.717194 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:32.717199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:32.717337 (+ 138us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.717445 (+ 108us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.717563 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":85,"threads_started":1}
I20240501 02:07:32.718254 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23477
I20240501 02:07:32.718336 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:32.718307 23606 heartbeater.cc:347] Connected to a master server at 127.22.12.126:33967
I20240501 02:07:32.718410 23606 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:32.718508 22577 external_mini_cluster.cc:1428] Setting key ee9ba77f326c8cb1d643a6499715b69b
I20240501 02:07:32.718679 23606 heartbeater.cc:511] Master 127.22.12.126:33967 requested a full tablet report, sending...
I20240501 02:07:32.719130 23146 ts_manager.cc:194] Registered new tserver with Master: 925c8e44c1104f8d96088bd92398afc0 (127.22.12.67:42509)
I20240501 02:07:32.719549 23146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:52147
I20240501 02:07:32.730479 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:07:32.737327 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.731992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.22.12.126:33967)
0501 02:07:32.732068 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.732069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.732158 (+ 89us) server_negotiation.cc:407] Connection header received
0501 02:07:32.732206 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.732207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.732216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.732271 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.733000 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.733504 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.734071 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.734189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.735193 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.735198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.735198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.735205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.736496 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.736753 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.736754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.736754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.736781 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.736886 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.736886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.736886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.737006 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.737077 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.737145 (+ 68us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.737229 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240501 02:07:32.739501 23146 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:32.740943 23146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:32.746135 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.743043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:41707 (local address 127.0.0.1:58966)
0501 02:07:32.743494 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.743501 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.743547 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.743756 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.743757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.743759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:32.743926 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.743927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.745253 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.745254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.745929 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.745931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.745994 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.745998 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.746025 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.746060 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20240501 02:07:32.746318 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.743037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38545 (local address 127.0.0.1:50394)
0501 02:07:32.743238 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.743247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.743313 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.743678 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.743679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.743680 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:32.743836 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.743837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.744835 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.744837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.745603 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.745605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.745677 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.745719 (+ 42us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.745772 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.745809 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":93,"threads_started":1}
I20240501 02:07:32.746443 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.743268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.22.12.65:38545)
0501 02:07:32.743419 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.743427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.743437 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:32.743475 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.743477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.743493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.743593 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:32.743924 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.744760 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.745718 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.746229 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.746286 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.746321 (+ 35us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.746369 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20240501 02:07:32.746470 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.743038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:42509 (local address 127.0.0.1:38216)
0501 02:07:32.743214 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.743239 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.743313 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.743615 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.743618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.743620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:32.743865 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.743867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.745265 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.745266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.746059 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.746061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.746129 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.746133 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.746159 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.746191 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20240501 02:07:32.747010 23272 tablet_service.cc:1467] Processing CreateTablet for tablet 26ccae36ce124630b66af1de4284d1dd (DEFAULT_TABLE table=test-table [id=241708ebe3024040859a8b9ab17c46c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:32.747110 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.743114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.22.12.66:41707)
0501 02:07:32.743454 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.743460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.743561 (+ 101us) server_negotiation.cc:407] Connection header received
0501 02:07:32.743601 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.743603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.743618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.743686 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:32.744000 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.745162 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.746400 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.746923 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.746967 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.746997 (+ 30us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.747034 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":76,"threads_started":1}
I20240501 02:07:32.747303 23272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ccae36ce124630b66af1de4284d1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:32.747454 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.743268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.22.12.67:42509)
0501 02:07:32.743419 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.743427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.743437 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:32.743475 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.743477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.743493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.743577 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:32.744075 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.745184 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.746743 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.747266 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.747314 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.747341 (+ 27us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.747384 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20240501 02:07:32.747617 23406 tablet_service.cc:1467] Processing CreateTablet for tablet 26ccae36ce124630b66af1de4284d1dd (DEFAULT_TABLE table=test-table [id=241708ebe3024040859a8b9ab17c46c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:32.747857 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ccae36ce124630b66af1de4284d1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:32.747975 23540 tablet_service.cc:1467] Processing CreateTablet for tablet 26ccae36ce124630b66af1de4284d1dd (DEFAULT_TABLE table=test-table [id=241708ebe3024040859a8b9ab17c46c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:32.748251 23540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ccae36ce124630b66af1de4284d1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:32.749902 23620 tablet_bootstrap.cc:492] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: Bootstrap starting.
I20240501 02:07:32.750563 23620 tablet_bootstrap.cc:654] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:32.750713 23621 tablet_bootstrap.cc:492] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: Bootstrap starting.
I20240501 02:07:32.751001 23622 tablet_bootstrap.cc:492] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0: Bootstrap starting.
I20240501 02:07:32.751194 23620 log.cc:824] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:32.751281 23621 tablet_bootstrap.cc:654] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:32.751484 23622 tablet_bootstrap.cc:654] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:32.751538 23621 log.cc:824] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:32.751838 23622 log.cc:824] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:32.751983 23620 tablet_bootstrap.cc:492] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: No bootstrap required, opened a new log
I20240501 02:07:32.752046 23620 ts_tablet_manager.cc:1367] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:07:32.752214 23621 tablet_bootstrap.cc:492] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: No bootstrap required, opened a new log
I20240501 02:07:32.752279 23621 ts_tablet_manager.cc:1367] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:32.752516 23622 tablet_bootstrap.cc:492] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0: No bootstrap required, opened a new log
I20240501 02:07:32.752589 23622 ts_tablet_manager.cc:1367] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:32.753944 23621 raft_consensus.cc:348] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.754086 23621 raft_consensus.cc:374] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:32.754108 23621 raft_consensus.cc:729] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ec6b6cc7674123ab7feddbefd280dd, State: Initialized, Role: FOLLOWER
I20240501 02:07:32.754215 23621 consensus_queue.cc:260] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.754442 23621 ts_tablet_manager.cc:1398] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:07:32.754568 23472 heartbeater.cc:503] Master 127.22.12.126:33967 was elected leader, sending a full tablet report...
I20240501 02:07:32.754604 23620 raft_consensus.cc:348] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.754773 23620 raft_consensus.cc:374] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:32.754798 23620 raft_consensus.cc:729] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5092a49cccec4cb8b2bee72b9e12fd3c, State: Initialized, Role: FOLLOWER
I20240501 02:07:32.754799 23622 raft_consensus.cc:348] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.754904 23622 raft_consensus.cc:374] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:32.754930 23622 raft_consensus.cc:729] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925c8e44c1104f8d96088bd92398afc0, State: Initialized, Role: FOLLOWER
I20240501 02:07:32.754905 23620 consensus_queue.cc:260] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.755009 23622 consensus_queue.cc:260] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.755232 23622 ts_tablet_manager.cc:1398] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:07:32.755232 23620 ts_tablet_manager.cc:1398] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 02:07:32.755329 23338 heartbeater.cc:503] Master 127.22.12.126:33967 was elected leader, sending a full tablet report...
I20240501 02:07:32.755355 23606 heartbeater.cc:503] Master 127.22.12.126:33967 was elected leader, sending a full tablet report...
W20240501 02:07:32.817607 23473 tablet.cc:2369] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:32.871104 23628 raft_consensus.cc:482] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:32.871237 23628 raft_consensus.cc:504] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.871590 23628 leader_election.cc:288] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 925c8e44c1104f8d96088bd92398afc0 (127.22.12.67:42509), a6ec6b6cc7674123ab7feddbefd280dd (127.22.12.66:41707)
I20240501 02:07:32.873780 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.871789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:41707 (local address 127.22.12.65:56095)
0501 02:07:32.871862 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.871874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.871919 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.872094 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.872095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.872096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:32.872306 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.872308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.873015 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.873016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.873597 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.873599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.873655 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.873659 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.873675 (+ 16us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.873706 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20240501 02:07:32.873907 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.871801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:42509 (local address 127.22.12.65:56009)
0501 02:07:32.872008 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.872018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:32.872080 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:32.872211 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:32.872212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:32.872213 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:32.872445 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.872447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.873070 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.873070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:32.873637 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:32.873639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.873709 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.873712 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:32.873735 (+ 23us) client_negotiation.cc:240] Negotiation successful
0501 02:07:32.873770 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":13,"thread_start_us":100,"threads_started":1}
I20240501 02:07:32.874269 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.871856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:56095 (local address 127.22.12.66:41707)
0501 02:07:32.871920 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.871922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.871931 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:07:32.871989 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.871991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.872001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.872079 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:32.872384 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.872927 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.873697 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.874175 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.874183 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.874203 (+ 20us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.874224 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 02:07:32.874297 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.871856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:56009 (local address 127.22.12.67:42509)
0501 02:07:32.871899 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.871901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.872106 (+ 205us) server_negotiation.cc:407] Connection header received
0501 02:07:32.872134 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.872135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.872145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.872200 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:32.872520 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.872970 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.873749 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.874190 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.874198 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.874220 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.874251 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:07:32.874581 23426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ccae36ce124630b66af1de4284d1dd" candidate_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" is_pre_election: true
I20240501 02:07:32.874591 23560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ccae36ce124630b66af1de4284d1dd" candidate_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925c8e44c1104f8d96088bd92398afc0" is_pre_election: true
I20240501 02:07:32.874717 23426 raft_consensus.cc:2454] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5092a49cccec4cb8b2bee72b9e12fd3c in term 0.
I20240501 02:07:32.874718 23560 raft_consensus.cc:2454] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5092a49cccec4cb8b2bee72b9e12fd3c in term 0.
I20240501 02:07:32.874926 23225 leader_election.cc:302] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5092a49cccec4cb8b2bee72b9e12fd3c, 925c8e44c1104f8d96088bd92398afc0; no voters:
I20240501 02:07:32.875059 23628 raft_consensus.cc:2781] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:32.875116 23628 raft_consensus.cc:482] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:32.875134 23628 raft_consensus.cc:3037] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:32.875778 23628 raft_consensus.cc:504] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.875907 23628 leader_election.cc:288] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [CANDIDATE]: Term 1 election: Requested vote from peers 925c8e44c1104f8d96088bd92398afc0 (127.22.12.67:42509), a6ec6b6cc7674123ab7feddbefd280dd (127.22.12.66:41707)
I20240501 02:07:32.876073 23560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ccae36ce124630b66af1de4284d1dd" candidate_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925c8e44c1104f8d96088bd92398afc0"
I20240501 02:07:32.876135 23426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ccae36ce124630b66af1de4284d1dd" candidate_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ec6b6cc7674123ab7feddbefd280dd"
I20240501 02:07:32.876161 23560 raft_consensus.cc:3037] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:32.876211 23426 raft_consensus.cc:3037] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:32.876999 23560 raft_consensus.cc:2454] T 26ccae36ce124630b66af1de4284d1dd P 925c8e44c1104f8d96088bd92398afc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5092a49cccec4cb8b2bee72b9e12fd3c in term 1.
I20240501 02:07:32.876999 23426 raft_consensus.cc:2454] T 26ccae36ce124630b66af1de4284d1dd P a6ec6b6cc7674123ab7feddbefd280dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5092a49cccec4cb8b2bee72b9e12fd3c in term 1.
I20240501 02:07:32.877168 23225 leader_election.cc:302] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5092a49cccec4cb8b2bee72b9e12fd3c, 925c8e44c1104f8d96088bd92398afc0; no voters:
I20240501 02:07:32.877259 23628 raft_consensus.cc:2781] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:32.877411 23628 raft_consensus.cc:686] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [term 1 LEADER]: Becoming Leader. State: Replica: 5092a49cccec4cb8b2bee72b9e12fd3c, State: Running, Role: LEADER
I20240501 02:07:32.877497 23628 consensus_queue.cc:237] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } }
I20240501 02:07:32.878259 23146 catalog_manager.cc:5484] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5092a49cccec4cb8b2bee72b9e12fd3c (127.22.12.65). New cstate: current_term: 1 leader_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925c8e44c1104f8d96088bd92398afc0" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5092a49cccec4cb8b2bee72b9e12fd3c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6ec6b6cc7674123ab7feddbefd280dd" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41707 } health_report { overall_health: UNKNOWN } } }
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:32.901615 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.895105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.22.12.65:38545)
0501 02:07:32.895171 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.895173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.895237 (+ 64us) server_negotiation.cc:407] Connection header received
0501 02:07:32.895323 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.895324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.895334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.895381 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.896043 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.896513 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.897042 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.897215 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.899185 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.899196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.899196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.899202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.900716 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.901031 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.901035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.901035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.901079 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.901202 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.901203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.901203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.901296 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.901420 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.901446 (+ 26us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.901518 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20240501 02:07:32.918017 23339 tablet.cc:2369] T 26ccae36ce124630b66af1de4284d1dd P 5092a49cccec4cb8b2bee72b9e12fd3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:32.921730 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.915721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.22.12.65:38545)
0501 02:07:32.915771 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.915773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.915798 (+ 25us) server_negotiation.cc:407] Connection header received
0501 02:07:32.915866 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.915867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.915876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.915925 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.916546 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.917030 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.917545 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.917663 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.919603 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.919608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.919609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.919615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.921029 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.921267 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.921268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.921268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.921286 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.921375 (+ 89us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.921375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.921375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.921450 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.921514 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.921607 (+ 93us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.921664 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:07:32 dist-test-slave-9tkh krb5kdc[23102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:32.942291 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:32.935975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.22.12.65:38545)
0501 02:07:32.936035 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:32.936037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:32.936100 (+ 63us) server_negotiation.cc:407] Connection header received
0501 02:07:32.936177 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:32.936177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:32.936186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.936233 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:32.936838 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.937286 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.937896 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.937997 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.940033 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:32.940037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:32.940038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:32.940044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:32.941492 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.941790 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.941791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.941791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.941809 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:32.941923 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:32.941923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:32.941924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:32.942002 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:32.942073 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:32.942156 (+ 83us) server_negotiation.cc:299] Negotiation successful
0501 02:07:32.942219 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20240501 02:07:32.942802 23252 tablet_service.cc:2908] Rejecting scan request for tablet 26ccae36ce124630b66af1de4284d1dd: Uninitialized: clean time has not yet been initialized
I20240501 02:07:32.943589 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23209
I20240501 02:07:32.948697 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23343
I20240501 02:07:32.952750 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23477
I20240501 02:07:32.956565 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23115
2024-05-01T02:07:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1828 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:33 dist-test-slave-9tkh krb5kdc[23644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:33 dist-test-slave-9tkh krb5kdc[23644](info): set up 2 sockets
May 01 02:07:33 dist-test-slave-9tkh krb5kdc[23644](info): commencing operation
krb5kdc: starting...
W20240501 02:07:33.861904 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.885s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:33 dist-test-slave-9tkh krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:33.973753 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39771
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:34.051024 23657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:34.051224 23657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:34.051255 23657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:34.052616 23657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:34.052662 23657 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240501 02:07:34.052680 23657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:34.052696 23657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:34.052709 23657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:34.052726 23657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:34.054317 23657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:39771
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:34.054558 23657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:34.054733 23657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:34.059103 23663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.059514 23665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.060029 23662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:34.060334 23657 server_base.cc:1034] running on GCE node
I20240501 02:07:34.060547 23657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:34.060739 23657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:34.061867 23657 hybrid_clock.cc:648] HybridClock initialized: now 1714529254061840 us; error 38 us; skew 500 ppm
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:34.064191 23657 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:34.064584 23657 webserver.cc:460] Webserver started at http://127.22.12.126:34407/ using document root <none> and password file <none>
I20240501 02:07:34.064738 23657 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:34.064778 23657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:34.064878 23657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:34.065814 23657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "d6534925d37347cf8458b1c41e1af593"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "7b6d7e71e7e43830e3158330802934be"
server_key_iv: "14c04f9a7710b88b5bb420ccd35e7ca8"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.066094 23657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "d6534925d37347cf8458b1c41e1af593"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "7b6d7e71e7e43830e3158330802934be"
server_key_iv: "14c04f9a7710b88b5bb420ccd35e7ca8"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.067207 23657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:34.067809 23672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.067957 23657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:34.068030 23657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "d6534925d37347cf8458b1c41e1af593"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "7b6d7e71e7e43830e3158330802934be"
server_key_iv: "14c04f9a7710b88b5bb420ccd35e7ca8"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.068084 23657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:34.091604 23657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:34.093714 23657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:34.093832 23657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:34.097991 23657 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:39771
I20240501 02:07:34.098039 23734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:39771 every 8 connection(s)
I20240501 02:07:34.098341 23657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:34.098935 23735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.100965 23735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: Bootstrap starting.
I20240501 02:07:34.101483 23735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.101718 23735 log.cc:824] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:34.102339 23735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: No bootstrap required, opened a new log
I20240501 02:07:34.103760 23735 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6534925d37347cf8458b1c41e1af593" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39771 } }
I20240501 02:07:34.103883 23735 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.103899 23735 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6534925d37347cf8458b1c41e1af593, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.104049 23735 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6534925d37347cf8458b1c41e1af593" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39771 } }
I20240501 02:07:34.104107 23735 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:34.104128 23735 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:34.104179 23735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.105026 23735 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6534925d37347cf8458b1c41e1af593" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39771 } }
I20240501 02:07:34.105118 23735 leader_election.cc:302] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6534925d37347cf8458b1c41e1af593; no voters:
I20240501 02:07:34.105262 23735 leader_election.cc:288] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:34.105305 23738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:34.105449 23738 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [term 1 LEADER]: Becoming Leader. State: Replica: d6534925d37347cf8458b1c41e1af593, State: Running, Role: LEADER
I20240501 02:07:34.105490 23735 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:34.105543 23738 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6534925d37347cf8458b1c41e1af593" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39771 } }
I20240501 02:07:34.105863 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6534925d37347cf8458b1c41e1af593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6534925d37347cf8458b1c41e1af593" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39771 } } }
I20240501 02:07:34.105885 23740 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6534925d37347cf8458b1c41e1af593. Latest consensus state: current_term: 1 leader_uuid: "d6534925d37347cf8458b1c41e1af593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6534925d37347cf8458b1c41e1af593" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39771 } } }
I20240501 02:07:34.105964 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:34.105970 23740 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:34.106182 23747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:34.106642 23747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:34.108150 23747 catalog_manager.cc:1347] Generated new cluster ID: 35101ebd85234df392ec5f98d0eaf989
I20240501 02:07:34.108203 23747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:34.108192 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23657
I20240501 02:07:34.108283 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:34.108428 22577 external_mini_cluster.cc:1428] Setting key 5147545bcdce121ac93fa91aaa031e94
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:34.117645 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.108843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.22.12.126:39771)
0501 02:07:34.109000 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.109009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.110473 (+ 1464us) server_negotiation.cc:407] Connection header received
0501 02:07:34.110608 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.110613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.110653 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.110764 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.111437 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.112175 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.112711 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.112928 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.114875 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.114881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.114881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.114897 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.116415 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.116705 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.116708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.116709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.116759 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.116895 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.116896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.116896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.117089 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.117213 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.117325 (+ 112us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.117448 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20240501 02:07:34.120666 23747 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:34.121363 23747 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:34.127645 23747 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: Generated new TSK 0
I20240501 02:07:34.127844 23747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:34.161901 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39771
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:34.241053 23761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:34.241247 23761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:34.241271 23761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:34.242656 23761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:34.242707 23761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:34.242725 23761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:07:34.242743 23761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:34.244400 23761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:34.244612 23761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:34.244804 23761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:34.249521 23767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.249948 23769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.250294 23766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:34.250731 23761 server_base.cc:1034] running on GCE node
I20240501 02:07:34.250921 23761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:34.251137 23761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:34.252275 23761 hybrid_clock.cc:648] HybridClock initialized: now 1714529254252257 us; error 36 us; skew 500 ppm
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:34.254966 23761 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:34.255400 23761 webserver.cc:460] Webserver started at http://127.22.12.65:39551/ using document root <none> and password file <none>
I20240501 02:07:34.255566 23761 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:34.255607 23761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:34.255712 23761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:34.256698 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "52391792cc8d4cc6ad9da16f00484a57"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "dfd7ac1aeb7438b3466b2c764571add6"
server_key_iv: "0763adf442bb13cc2e1a18717f1571c0"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.256999 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "52391792cc8d4cc6ad9da16f00484a57"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "dfd7ac1aeb7438b3466b2c764571add6"
server_key_iv: "0763adf442bb13cc2e1a18717f1571c0"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.258173 23761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:07:34.258924 23776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.259106 23761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:07:34.259191 23761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "52391792cc8d4cc6ad9da16f00484a57"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "dfd7ac1aeb7438b3466b2c764571add6"
server_key_iv: "0763adf442bb13cc2e1a18717f1571c0"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.259250 23761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:34.287323 23761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:34.289054 23761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:34.289184 23761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:34.289732 23761 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:34.289772 23761 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.289801 23761 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:34.289816 23761 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.296190 23761 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:35557
I20240501 02:07:34.296255 23891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:35557 every 8 connection(s)
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:34.296536 23761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:34.299472 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.290134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:53599 (local address 127.22.12.126:39771)
0501 02:07:34.290202 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.290205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.290430 (+ 225us) server_negotiation.cc:407] Connection header received
0501 02:07:34.291139 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.291140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.291154 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.291224 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.292256 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.293020 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.293796 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.293975 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.296969 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.296974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.296975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.296982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.298217 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.298637 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.298638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.298639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.298661 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.298830 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.298831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.298832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.298933 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.299015 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.299259 (+ 244us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.299360 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 02:07:34.299559 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.290113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.22.12.65:53599)
0501 02:07:34.290297 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.290331 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.291018 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.291288 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.291293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.291737 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.292152 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.292158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.293133 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.293134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.293706 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.293709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.293856 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.294435 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.294453 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.296874 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:34.298315 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.298321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.298324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.298562 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.298712 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.298713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.298713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.298772 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.299006 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:34.299011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:34.299130 (+ 119us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.299218 (+ 88us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.299362 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20240501 02:07:34.300176 23892 heartbeater.cc:347] Connected to a master server at 127.22.12.126:39771
I20240501 02:07:34.300264 23892 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:34.300505 23892 heartbeater.cc:511] Master 127.22.12.126:39771 requested a full tablet report, sending...
I20240501 02:07:34.301038 23688 ts_manager.cc:194] Registered new tserver with Master: 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557)
I20240501 02:07:34.301949 23688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:53599
I20240501 02:07:34.306176 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.301305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:58861 (local address 127.22.12.126:39771)
0501 02:07:34.301339 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.301342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.301496 (+ 154us) server_negotiation.cc:407] Connection header received
0501 02:07:34.301522 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.301522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.301532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.301586 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.302123 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.302797 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.303340 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.303470 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.304100 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.304104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.304104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.304111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.305392 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.305674 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.305675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.305675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.305695 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.305800 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.305800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.305800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.305876 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.305915 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.306047 (+ 132us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.306110 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20240501 02:07:34.306200 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.301239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.22.12.65:58861)
0501 02:07:34.301404 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.301414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.301481 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.301596 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.301597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.301750 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.301992 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.301994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.302876 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.302876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.303254 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.303256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.303314 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.303768 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.303771 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.304015 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:34.305457 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.305457 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.305458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.305584 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.305739 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.305740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.305740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.305764 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.305929 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:34.305930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:34.305969 (+ 39us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.306024 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.306118 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20240501 02:07:34.306495 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23761
I20240501 02:07:34.306627 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:34.306803 22577 external_mini_cluster.cc:1428] Setting key f5fd8630c15e12996c41065c6f5b87fc
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:34.329618 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39771
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:34.408201 23901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:34.408391 23901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:34.408416 23901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:34.409755 23901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:34.409803 23901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:34.409822 23901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:07:34.409854 23901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:34.411512 23901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:34.411728 23901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:34.411916 23901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:34.416168 23909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.416643 23906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.417068 23907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:34.417536 23901 server_base.cc:1034] running on GCE node
I20240501 02:07:34.417744 23901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:34.417953 23901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:34.419075 23901 hybrid_clock.cc:648] HybridClock initialized: now 1714529254419062 us; error 30 us; skew 500 ppm
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:34.421368 23901 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:34.421788 23901 webserver.cc:460] Webserver started at http://127.22.12.66:43255/ using document root <none> and password file <none>
I20240501 02:07:34.421945 23901 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:34.421980 23901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:34.422060 23901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:34.423069 23901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "3422e6ff77464cd0aa0b1daa30ae37ce"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "7716abeeeb5935d228d37d8f7d416d12"
server_key_iv: "eda70af15cc0316f4220587be5972d49"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.423331 23901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "3422e6ff77464cd0aa0b1daa30ae37ce"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "7716abeeeb5935d228d37d8f7d416d12"
server_key_iv: "eda70af15cc0316f4220587be5972d49"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.424435 23901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:34.425065 23916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.425209 23901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:34.425283 23901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "3422e6ff77464cd0aa0b1daa30ae37ce"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "7716abeeeb5935d228d37d8f7d416d12"
server_key_iv: "eda70af15cc0316f4220587be5972d49"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.425340 23901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:34.445415 23901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:34.447327 23901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:34.447443 23901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:34.448006 23901 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:34.448051 23901 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.448099 23901 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:34.448117 23901 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:34.454214 23901 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:35787
I20240501 02:07:34.454272 24031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:35787 every 8 connection(s)
I20240501 02:07:34.454597 23901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:34.456806 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.448773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:57819 (local address 127.22.12.126:39771)
0501 02:07:34.448835 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.448837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.448847 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:34.449372 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.449373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.449387 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.449466 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.450442 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.451212 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.451816 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.451984 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.454601 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.454606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.454606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.454613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.455856 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.456193 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.456194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.456194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.456216 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.456354 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.456355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.456355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.456449 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.456496 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.456678 (+ 182us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.456744 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:07:34.456890 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.448531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.22.12.66:57819)
0501 02:07:34.448733 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.448758 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.449246 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.449536 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.449541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.449929 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.450332 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.450338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.451293 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.451294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.451736 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.451738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.451866 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.452369 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.452386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.454477 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:34.455920 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.455924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.455926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.456126 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.456269 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.456269 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.456270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.456314 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.456491 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:34.456493 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:34.456587 (+ 94us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.456663 (+ 76us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.456755 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20240501 02:07:34.457343 24032 heartbeater.cc:347] Connected to a master server at 127.22.12.126:39771
I20240501 02:07:34.457422 24032 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:34.457625 24032 heartbeater.cc:511] Master 127.22.12.126:39771 requested a full tablet report, sending...
I20240501 02:07:34.458016 23688 ts_manager.cc:194] Registered new tserver with Master: 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787)
I20240501 02:07:34.458485 23688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:57819
I20240501 02:07:34.462592 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.458363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:39021 (local address 127.22.12.126:39771)
0501 02:07:34.458400 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.458402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.458466 (+ 64us) server_negotiation.cc:407] Connection header received
0501 02:07:34.458554 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.458555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.458564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.458633 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.459055 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.459497 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.459980 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.460134 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.460664 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.460667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.460667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.460673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.461672 (+ 999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.461939 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.461940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.461940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.461963 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.462120 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.462121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.462121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.462224 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.462276 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.462408 (+ 132us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.462476 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:07:34.462605 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.458294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.22.12.66:39021)
0501 02:07:34.458402 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.458412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.458480 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.458693 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.458694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.458811 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.458982 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.458983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.459562 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.459563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.459913 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.459914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.459971 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.460377 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.460380 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.460588 (+ 208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:34.461743 (+ 1155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.461744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.461745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.461866 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.462029 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.462030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.462030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.462058 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.462275 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:34.462276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:34.462316 (+ 40us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.462375 (+ 59us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.462467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20240501 02:07:34.464048 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 23901
I20240501 02:07:34.464144 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:34.464397 22577 external_mini_cluster.cc:1428] Setting key 5d3c81c4c1731ff802f957a5576b4738
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:34.489256 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39771
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:34.567878 24041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:34.568063 24041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:34.568084 24041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:34.569314 24041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:34.569350 24041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:34.569368 24041 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:07:34.569396 24041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:34.571010 24041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:34.571195 24041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:34.571363 24041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:34.575645 24049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.575855 24047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:34.576285 24046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:34.576656 24041 server_base.cc:1034] running on GCE node
I20240501 02:07:34.576828 24041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:34.577034 24041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:34.578172 24041 hybrid_clock.cc:648] HybridClock initialized: now 1714529254578154 us; error 32 us; skew 500 ppm
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:34.580490 24041 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:34.580886 24041 webserver.cc:460] Webserver started at http://127.22.12.67:38509/ using document root <none> and password file <none>
I20240501 02:07:34.581055 24041 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:34.581094 24041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:34.581193 24041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:34.582123 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "04fa33e914c54ba1bdbc6d45695ac64b"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "86b58a486a2e52dea01581325ede3bb7"
server_key_iv: "8f99b72de17e7e2da70a94d3e7aa518e"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.582409 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "04fa33e914c54ba1bdbc6d45695ac64b"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "86b58a486a2e52dea01581325ede3bb7"
server_key_iv: "8f99b72de17e7e2da70a94d3e7aa518e"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.583652 24041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:07:34.584300 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.584455 24041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:07:34.584528 24041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "04fa33e914c54ba1bdbc6d45695ac64b"
format_stamp: "Formatted at 2024-05-01 02:07:34 on dist-test-slave-9tkh"
server_key: "86b58a486a2e52dea01581325ede3bb7"
server_key_iv: "8f99b72de17e7e2da70a94d3e7aa518e"
server_key_version: "encryptionkey@0"
I20240501 02:07:34.584594 24041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:34.595839 24041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:34.597851 24041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:34.597961 24041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:34.598419 24041 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:34.598459 24041 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.598488 24041 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:34.598505 24041 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:34.604244 24041 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:36483
I20240501 02:07:34.604337 24171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:36483 every 8 connection(s)
I20240501 02:07:34.604610 24041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:34.607170 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.598838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:52119 (local address 127.22.12.126:39771)
0501 02:07:34.598891 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.598893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.599070 (+ 177us) server_negotiation.cc:407] Connection header received
0501 02:07:34.599595 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.599596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.599606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.599638 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.600726 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.601476 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.602212 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.602405 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.604646 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.604657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.604657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.604665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.606079 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.606443 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.606444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.606444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.606471 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.606656 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.606657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.606658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.606756 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.606807 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.607035 (+ 228us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.607102 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:07:34.607359 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.598843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.22.12.67:52119)
0501 02:07:34.598967 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.598993 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.599473 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.599801 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.599805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.600183 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.600636 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.600641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.601561 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.601562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.602120 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.602122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.602259 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.602813 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.602828 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.604553 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:34.606164 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.606168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.606170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.606353 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.606521 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.606521 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.606522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.606600 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.606806 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:34.606810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:34.606915 (+ 105us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.607025 (+ 110us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.607189 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20240501 02:07:34.607870 24172 heartbeater.cc:347] Connected to a master server at 127.22.12.126:39771
I20240501 02:07:34.607959 24172 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:34.608141 24172 heartbeater.cc:511] Master 127.22.12.126:39771 requested a full tablet report, sending...
I20240501 02:07:34.608515 23689 ts_manager.cc:194] Registered new tserver with Master: 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483)
I20240501 02:07:34.608985 23689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:52119
I20240501 02:07:34.613031 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.608879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:41713 (local address 127.22.12.126:39771)
0501 02:07:34.608925 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.608927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.609097 (+ 170us) server_negotiation.cc:407] Connection header received
0501 02:07:34.609123 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.609123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.609134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.609196 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.609586 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.610041 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.610524 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.610701 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.611236 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.611239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.611239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.611245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.612218 (+ 973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.612471 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.612472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.612472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.612492 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.612609 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.612610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.612610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.612688 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.612738 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.612863 (+ 125us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.612953 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:07:34.613041 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.608812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.22.12.67:41713)
0501 02:07:34.608981 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.608991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.609055 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.609202 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.609203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.609338 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.609517 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.609519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.610105 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.610106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.610460 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.610461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.610519 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.610954 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.610958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.611164 (+ 206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:34.612287 (+ 1123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.612288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.612289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.612416 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.612543 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:34.612543 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:34.612544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:34.612571 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:34.612735 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:34.612736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:34.612776 (+ 40us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.612835 (+ 59us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.612928 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20240501 02:07:34.613751 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 24041
I20240501 02:07:34.613853 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:34.614030 22577 external_mini_cluster.cc:1428] Setting key ac9fa062400478f48a3fab1874f4119d
I20240501 02:07:34.614974 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:34.634442 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.628625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.22.12.126:39771)
0501 02:07:34.628686 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.628688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.628714 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:07:34.628741 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.628742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.628751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.628789 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.629436 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.629912 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.630466 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.630658 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.632473 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.632477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.632477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.632484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:34.633531 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.633799 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.633800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.633801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.633824 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:34.633952 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:34.633953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:34.633953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:34.634044 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:34.634135 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.634258 (+ 123us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.634344 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:07:34.636085 23689 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:34.637529 23689 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:34.642892 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.639773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:36483 (local address 127.0.0.1:49048)
0501 02:07:34.640027 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.640037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.640094 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.640363 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.640364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.640366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.640538 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.640539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.641877 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.641878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.642597 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.642599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.642657 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.642703 (+ 46us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.642754 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.642796 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":55,"threads_started":1}
I20240501 02:07:34.643139 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.639645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.22.12.65:35557)
0501 02:07:34.639803 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.639811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.640096 (+ 285us) server_negotiation.cc:407] Connection header received
0501 02:07:34.640158 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.640160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.640174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.640240 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.640657 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.641386 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.642310 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.642824 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.642902 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.643022 (+ 120us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.643071 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20240501 02:07:34.643208 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.639571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:35787 (local address 127.0.0.1:46978)
0501 02:07:34.639732 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.639761 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.639831 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.640236 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.640238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.640241 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.640506 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.640508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.641843 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.641843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.642760 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.642762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.642827 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.642830 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.642877 (+ 47us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.642923 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20240501 02:07:34.643345 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.639572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35557 (local address 127.0.0.1:59316)
0501 02:07:34.640007 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.640017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.640095 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.640337 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.640338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.640340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.640554 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.640557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.641461 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.641462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.642195 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.642197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.642929 (+ 732us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.642932 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.643254 (+ 322us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.643296 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":5,"thread_start_us":48,"threads_started":1}
I20240501 02:07:34.643366 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.639854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.22.12.67:36483)
0501 02:07:34.640019 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.640027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.640148 (+ 121us) server_negotiation.cc:407] Connection header received
0501 02:07:34.640187 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.640190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.640206 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.640274 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.640618 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.641767 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.642689 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.643178 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.643223 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.643258 (+ 35us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.643300 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20240501 02:07:34.643949 24106 tablet_service.cc:1467] Processing CreateTablet for tablet 99371c47a06b4ceb9c720935be1280db (DEFAULT_TABLE table=test-table [id=1ade7d07c1354d1a86827af76a4b6e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:34.644064 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.639896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.22.12.66:35787)
0501 02:07:34.640054 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.640060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.640070 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:34.640109 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.640111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.640129 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.640224 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.640588 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.641762 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.643370 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.643870 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.643924 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.643957 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.643995 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20240501 02:07:34.644280 24106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99371c47a06b4ceb9c720935be1280db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.644598 23966 tablet_service.cc:1467] Processing CreateTablet for tablet 99371c47a06b4ceb9c720935be1280db (DEFAULT_TABLE table=test-table [id=1ade7d07c1354d1a86827af76a4b6e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:34.644842 23966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99371c47a06b4ceb9c720935be1280db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.646922 24195 tablet_bootstrap.cc:492] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Bootstrap starting.
I20240501 02:07:34.646898 23826 tablet_service.cc:1467] Processing CreateTablet for tablet 99371c47a06b4ceb9c720935be1280db (DEFAULT_TABLE table=test-table [id=1ade7d07c1354d1a86827af76a4b6e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:34.647161 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99371c47a06b4ceb9c720935be1280db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.647215 24196 tablet_bootstrap.cc:492] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Bootstrap starting.
I20240501 02:07:34.647821 24195 tablet_bootstrap.cc:654] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.647933 24196 tablet_bootstrap.cc:654] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.648145 24195 log.cc:824] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:34.648181 24196 log.cc:824] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:34.648893 24196 tablet_bootstrap.cc:492] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: No bootstrap required, opened a new log
I20240501 02:07:34.648958 24196 ts_tablet_manager.cc:1367] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:34.649185 24195 tablet_bootstrap.cc:492] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: No bootstrap required, opened a new log
I20240501 02:07:34.649246 24195 ts_tablet_manager.cc:1367] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 02:07:34.650666 24199 tablet_bootstrap.cc:492] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Bootstrap starting.
I20240501 02:07:34.650701 24196 raft_consensus.cc:348] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.650838 24196 raft_consensus.cc:374] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.650861 24196 raft_consensus.cc:729] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3422e6ff77464cd0aa0b1daa30ae37ce, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.650931 24195 raft_consensus.cc:348] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.650972 24196 consensus_queue.cc:260] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.651033 24195 raft_consensus.cc:374] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.651055 24195 raft_consensus.cc:729] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fa33e914c54ba1bdbc6d45695ac64b, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.651150 24195 consensus_queue.cc:260] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.651202 24196 ts_tablet_manager.cc:1398] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 02:07:34.651330 24032 heartbeater.cc:503] Master 127.22.12.126:39771 was elected leader, sending a full tablet report...
I20240501 02:07:34.651350 24195 ts_tablet_manager.cc:1398] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:07:34.651484 24172 heartbeater.cc:503] Master 127.22.12.126:39771 was elected leader, sending a full tablet report...
I20240501 02:07:34.651490 24199 tablet_bootstrap.cc:654] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.651754 24199 log.cc:824] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:34.652411 24199 tablet_bootstrap.cc:492] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: No bootstrap required, opened a new log
I20240501 02:07:34.652473 24199 ts_tablet_manager.cc:1367] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:34.653642 24199 raft_consensus.cc:348] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.653745 24199 raft_consensus.cc:374] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.653767 24199 raft_consensus.cc:729] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52391792cc8d4cc6ad9da16f00484a57, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.653849 24199 consensus_queue.cc:260] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.654054 24199 ts_tablet_manager.cc:1398] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:07:34.654116 23892 heartbeater.cc:503] Master 127.22.12.126:39771 was elected leader, sending a full tablet report...
W20240501 02:07:34.705085 24033 tablet.cc:2369] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:07:34.797350 23893 tablet.cc:2369] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:07:34.855453 24173 tablet.cc:2369] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:34.870003 24201 raft_consensus.cc:482] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:34.870127 24201 raft_consensus.cc:504] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.870466 24201 leader_election.cc:288] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787), 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557)
I20240501 02:07:34.872599 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.870703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35557 (local address 127.22.12.67:52831)
0501 02:07:34.870744 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.870756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.870798 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.870972 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.870973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.870975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.871167 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.871169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.871818 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.871819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.872390 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.872391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.872458 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.872463 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.872488 (+ 25us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.872526 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20240501 02:07:34.872982 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.870703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:35787 (local address 127.22.12.67:50411)
0501 02:07:34.871011 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.871019 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.871070 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.871179 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.871180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.871181 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.871441 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.871443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.872110 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.872111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.872728 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.872731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.872833 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.872837 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.872864 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.872899 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":138,"thread_start_us":107,"threads_started":1}
I20240501 02:07:34.873046 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.870755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:52831 (local address 127.22.12.65:35557)
0501 02:07:34.870819 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.870821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.870836 (+ 15us) server_negotiation.cc:407] Connection header received
0501 02:07:34.870882 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.870883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.870890 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.870952 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.871239 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.871746 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.872469 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.872924 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.872932 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.872953 (+ 21us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.872983 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 02:07:34.873397 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.870755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:50411 (local address 127.22.12.66:35787)
0501 02:07:34.870831 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.870833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.871076 (+ 243us) server_negotiation.cc:407] Connection header received
0501 02:07:34.871120 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.871121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.871130 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.871170 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.871511 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.872035 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.872861 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.873267 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.873275 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.873300 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.873327 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 02:07:34.873363 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99371c47a06b4ceb9c720935be1280db" candidate_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52391792cc8d4cc6ad9da16f00484a57" is_pre_election: true
I20240501 02:07:34.873512 23846 raft_consensus.cc:2454] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fa33e914c54ba1bdbc6d45695ac64b in term 0.
I20240501 02:07:34.873678 23986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99371c47a06b4ceb9c720935be1280db" candidate_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" is_pre_election: true
I20240501 02:07:34.873800 23986 raft_consensus.cc:2454] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fa33e914c54ba1bdbc6d45695ac64b in term 0.
I20240501 02:07:34.874011 24059 leader_election.cc:302] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fa33e914c54ba1bdbc6d45695ac64b, 3422e6ff77464cd0aa0b1daa30ae37ce; no voters:
I20240501 02:07:34.874174 24201 raft_consensus.cc:2781] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:34.874218 24201 raft_consensus.cc:482] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:34.874231 24201 raft_consensus.cc:3037] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.874888 24201 raft_consensus.cc:504] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.875012 24201 leader_election.cc:288] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [CANDIDATE]: Term 1 election: Requested vote from peers 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787), 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557)
I20240501 02:07:34.875178 23986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99371c47a06b4ceb9c720935be1280db" candidate_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce"
I20240501 02:07:34.875231 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99371c47a06b4ceb9c720935be1280db" candidate_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52391792cc8d4cc6ad9da16f00484a57"
I20240501 02:07:34.875264 23986 raft_consensus.cc:3037] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.875291 23846 raft_consensus.cc:3037] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.875919 23986 raft_consensus.cc:2454] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fa33e914c54ba1bdbc6d45695ac64b in term 1.
I20240501 02:07:34.875991 23846 raft_consensus.cc:2454] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fa33e914c54ba1bdbc6d45695ac64b in term 1.
I20240501 02:07:34.876104 24059 leader_election.cc:302] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fa33e914c54ba1bdbc6d45695ac64b, 3422e6ff77464cd0aa0b1daa30ae37ce; no voters:
I20240501 02:07:34.876217 24201 raft_consensus.cc:2781] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:34.876358 24201 raft_consensus.cc:686] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 LEADER]: Becoming Leader. State: Replica: 04fa33e914c54ba1bdbc6d45695ac64b, State: Running, Role: LEADER
I20240501 02:07:34.876452 24201 consensus_queue.cc:237] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.877283 23688 catalog_manager.cc:5484] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b reported cstate change: term changed from 0 to 1, leader changed from <none> to 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67). New cstate: current_term: 1 leader_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } health_report { overall_health: HEALTHY } } }
I20240501 02:07:34.925879 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.923587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.22.12.67:36483)
0501 02:07:34.923640 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.923642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.923650 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:07:34.923715 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.923715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.923734 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.923787 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:34.924096 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.924813 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.925435 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.925560 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.925593 (+ 33us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:34.925673 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:34.925729 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.925787 (+ 58us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.925808 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:07:34.927667 23846 raft_consensus.cc:1261] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Refusing update from remote peer 04fa33e914c54ba1bdbc6d45695ac64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:34.927685 23986 raft_consensus.cc:1261] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Refusing update from remote peer 04fa33e914c54ba1bdbc6d45695ac64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:34.927991 24201 consensus_queue.cc:1035] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:34.928138 24205 consensus_queue.cc:1035] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:34.930860 24210 mvcc.cc:205] Tried to move back new op lower bound from 7022711828180922368 to 7022711827974459392. Current Snapshot: MvccSnapshot[applied={T|T < 7022711828180922368}]
I20240501 02:07:34.930963 24209 mvcc.cc:205] Tried to move back new op lower bound from 7022711828180922368 to 7022711827974459392. Current Snapshot: MvccSnapshot[applied={T|T < 7022711828180922368}]
I20240501 02:07:34.933416 23688 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:34.933498 23688 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:34.934165 23688 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: Sending DeleteTablet for 3 replicas of tablet 99371c47a06b4ceb9c720935be1280db
I20240501 02:07:34.934495 23826 tablet_service.cc:1514] Processing DeleteTablet for tablet 99371c47a06b4ceb9c720935be1280db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:34 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:59316
I20240501 02:07:34.934515 23966 tablet_service.cc:1514] Processing DeleteTablet for tablet 99371c47a06b4ceb9c720935be1280db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:34 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:46978
I20240501 02:07:34.934515 24106 tablet_service.cc:1514] Processing DeleteTablet for tablet 99371c47a06b4ceb9c720935be1280db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:34 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:49048
I20240501 02:07:34.934844 24219 tablet_replica.cc:330] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: stopping tablet replica
I20240501 02:07:34.934927 24219 raft_consensus.cc:2229] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:34.935025 24219 raft_consensus.cc:2258] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:34.935017 23688 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 02:07:34.935170 24218 tablet_replica.cc:330] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: stopping tablet replica
I20240501 02:07:34.935245 24218 raft_consensus.cc:2229] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:34.935346 24218 raft_consensus.cc:2258] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 02:07:34.935317 23688 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:34.935676 24219 ts_tablet_manager.cc:1875] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:34.936699 24217 tablet_replica.cc:330] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: stopping tablet replica
I20240501 02:07:34.936789 24217 raft_consensus.cc:2229] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:34.936872 24217 raft_consensus.cc:2258] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:34.937487 24219 ts_tablet_manager.cc:1888] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:34.937556 24219 log.cc:1196] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/99371c47a06b4ceb9c720935be1280db
I20240501 02:07:34.937816 23966 tablet_service.cc:1467] Processing CreateTablet for tablet 302bedb9c0c5446bb816636bd47b1e24 (DEFAULT_TABLE table=test-table [id=fad3e6eb3b3c4639a62116a6d13425ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:34.937870 24219 ts_tablet_manager.cc:1909] T 99371c47a06b4ceb9c720935be1280db P 3422e6ff77464cd0aa0b1daa30ae37ce: Deleting consensus metadata
I20240501 02:07:34.937909 23966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302bedb9c0c5446bb816636bd47b1e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.937881 23826 tablet_service.cc:1467] Processing CreateTablet for tablet 302bedb9c0c5446bb816636bd47b1e24 (DEFAULT_TABLE table=test-table [id=fad3e6eb3b3c4639a62116a6d13425ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:34.937992 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302bedb9c0c5446bb816636bd47b1e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.938804 24196 tablet_bootstrap.cc:492] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Bootstrap starting.
I20240501 02:07:34.938899 23675 catalog_manager.cc:4830] TS 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787): tablet 99371c47a06b4ceb9c720935be1280db (table test-table [id=1ade7d07c1354d1a86827af76a4b6e3c]) successfully deleted
I20240501 02:07:34.939069 24106 tablet_service.cc:1467] Processing CreateTablet for tablet 302bedb9c0c5446bb816636bd47b1e24 (DEFAULT_TABLE table=test-table [id=fad3e6eb3b3c4639a62116a6d13425ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:34.939163 24106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302bedb9c0c5446bb816636bd47b1e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:34.939287 24196 tablet_bootstrap.cc:654] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.939286 24199 tablet_bootstrap.cc:492] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Bootstrap starting.
I20240501 02:07:34.939608 24199 tablet_bootstrap.cc:654] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.940207 24199 tablet_bootstrap.cc:492] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: No bootstrap required, opened a new log
I20240501 02:07:34.940246 24199 ts_tablet_manager.cc:1367] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:34.940239 24195 tablet_bootstrap.cc:492] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Bootstrap starting.
I20240501 02:07:34.940397 24199 raft_consensus.cc:348] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.940464 24199 raft_consensus.cc:374] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.940488 24199 raft_consensus.cc:729] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52391792cc8d4cc6ad9da16f00484a57, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.940524 24199 consensus_queue.cc:260] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.940600 24195 tablet_bootstrap.cc:654] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:34.940619 24199 ts_tablet_manager.cc:1398] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:34.940955 24196 tablet_bootstrap.cc:492] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: No bootstrap required, opened a new log
I20240501 02:07:34.941001 24196 ts_tablet_manager.cc:1367] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:34.941143 24196 raft_consensus.cc:348] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.941201 24196 raft_consensus.cc:374] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.941219 24196 raft_consensus.cc:729] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3422e6ff77464cd0aa0b1daa30ae37ce, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.941257 24196 consensus_queue.cc:260] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.941287 24195 tablet_bootstrap.cc:492] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: No bootstrap required, opened a new log
I20240501 02:07:34.941319 24195 ts_tablet_manager.cc:1367] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:34.941331 24196 ts_tablet_manager.cc:1398] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.941475 24195 raft_consensus.cc:348] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.941533 24195 raft_consensus.cc:374] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:34.941551 24195 raft_consensus.cc:729] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fa33e914c54ba1bdbc6d45695ac64b, State: Initialized, Role: FOLLOWER
I20240501 02:07:34.941572 24217 ts_tablet_manager.cc:1875] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:34.941582 24195 consensus_queue.cc:260] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.941661 24195 ts_tablet_manager.cc:1398] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:34.941987 24218 ts_tablet_manager.cc:1875] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:34.944772 24217 ts_tablet_manager.cc:1888] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:34.944839 24217 log.cc:1196] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/99371c47a06b4ceb9c720935be1280db
I20240501 02:07:34.945070 24217 ts_tablet_manager.cc:1909] T 99371c47a06b4ceb9c720935be1280db P 52391792cc8d4cc6ad9da16f00484a57: Deleting consensus metadata
I20240501 02:07:34.945293 24218 ts_tablet_manager.cc:1888] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:34.945358 24218 log.cc:1196] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/99371c47a06b4ceb9c720935be1280db
I20240501 02:07:34.945509 23674 catalog_manager.cc:4830] TS 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557): tablet 99371c47a06b4ceb9c720935be1280db (table test-table [id=1ade7d07c1354d1a86827af76a4b6e3c]) successfully deleted
I20240501 02:07:34.945680 24218 ts_tablet_manager.cc:1909] T 99371c47a06b4ceb9c720935be1280db P 04fa33e914c54ba1bdbc6d45695ac64b: Deleting consensus metadata
I20240501 02:07:34.946043 23675 catalog_manager.cc:4830] TS 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483): tablet 99371c47a06b4ceb9c720935be1280db (table test-table [id=1ade7d07c1354d1a86827af76a4b6e3c]) successfully deleted
I20240501 02:07:34.966665 24200 raft_consensus.cc:482] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:34.966774 24200 raft_consensus.cc:504] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.967108 24200 leader_election.cc:288] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557), 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483)
I20240501 02:07:34.969558 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.967270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35557 (local address 127.22.12.66:47007)
0501 02:07:34.967407 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.967418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.967472 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.967635 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.967636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.967638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.967824 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.967826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.968554 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.968555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.969317 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.969319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.969397 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.969402 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.969432 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.969468 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20240501 02:07:34.970012 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.967449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:47007 (local address 127.22.12.65:35557)
0501 02:07:34.967485 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.967487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.967497 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:34.967527 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.967528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.967537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.967599 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.967913 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.968476 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.969410 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.969875 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.969884 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.969905 (+ 21us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.969935 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:07:34.970216 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.967272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:36483 (local address 127.22.12.66:50273)
0501 02:07:34.967464 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.967472 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.967552 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.967688 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.967689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.967692 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:34.967930 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.967932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.968828 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.968829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.969839 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.969842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.969902 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.969914 (+ 12us) client_negotiation.cc:769] Sending connection context
0501 02:07:34.969941 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:07:34.969980 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":14,"thread_start_us":58,"threads_started":1}
I20240501 02:07:34.970199 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302bedb9c0c5446bb816636bd47b1e24" candidate_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52391792cc8d4cc6ad9da16f00484a57" is_pre_election: true
I20240501 02:07:34.970274 23846 raft_consensus.cc:2454] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3422e6ff77464cd0aa0b1daa30ae37ce in term 0.
I20240501 02:07:34.970428 23918 leader_election.cc:302] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3422e6ff77464cd0aa0b1daa30ae37ce, 52391792cc8d4cc6ad9da16f00484a57; no voters:
I20240501 02:07:34.970502 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.967370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:50273 (local address 127.22.12.67:36483)
0501 02:07:34.967411 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.967413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.967530 (+ 117us) server_negotiation.cc:407] Connection header received
0501 02:07:34.967620 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.967621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.967630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.967677 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:34.968032 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.968749 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.969919 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.970378 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.970386 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:34.970409 (+ 23us) server_negotiation.cc:299] Negotiation successful
0501 02:07:34.970442 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:07:34.970585 24200 raft_consensus.cc:2781] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:34.970628 24200 raft_consensus.cc:482] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:34.970646 24200 raft_consensus.cc:3037] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.970782 24126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302bedb9c0c5446bb816636bd47b1e24" candidate_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" is_pre_election: true
I20240501 02:07:34.970894 24126 raft_consensus.cc:2454] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3422e6ff77464cd0aa0b1daa30ae37ce in term 0.
I20240501 02:07:34.971485 24200 raft_consensus.cc:504] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.971635 24200 leader_election.cc:288] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [CANDIDATE]: Term 1 election: Requested vote from peers 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557), 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483)
I20240501 02:07:34.971820 24126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302bedb9c0c5446bb816636bd47b1e24" candidate_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fa33e914c54ba1bdbc6d45695ac64b"
I20240501 02:07:34.971839 23846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302bedb9c0c5446bb816636bd47b1e24" candidate_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52391792cc8d4cc6ad9da16f00484a57"
I20240501 02:07:34.971896 24126 raft_consensus.cc:3037] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.971899 23846 raft_consensus.cc:3037] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:34.972653 24126 raft_consensus.cc:2454] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3422e6ff77464cd0aa0b1daa30ae37ce in term 1.
I20240501 02:07:34.972656 23846 raft_consensus.cc:2454] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3422e6ff77464cd0aa0b1daa30ae37ce in term 1.
I20240501 02:07:34.972834 23918 leader_election.cc:302] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3422e6ff77464cd0aa0b1daa30ae37ce, 52391792cc8d4cc6ad9da16f00484a57; no voters:
I20240501 02:07:34.972944 24200 raft_consensus.cc:2781] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:34.973076 24200 raft_consensus.cc:686] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 LEADER]: Becoming Leader. State: Replica: 3422e6ff77464cd0aa0b1daa30ae37ce, State: Running, Role: LEADER
I20240501 02:07:34.973150 24200 consensus_queue.cc:237] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:34.973798 23689 catalog_manager.cc:5484] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66). New cstate: current_term: 1 leader_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } health_report { overall_health: UNKNOWN } } }
May 01 02:07:34 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:35.001964 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.995344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39771 (local address 127.0.0.1:56340)
0501 02:07:34.995479 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:34.995490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:34.995553 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:34.995697 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:34.995698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:34.996035 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:34.996221 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.996224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.997113 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.997113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:34.997476 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:34.997477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.997533 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.998031 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:34.998036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:34.999398 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:35.001120 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:35.001127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:35.001130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:35.001313 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:35.001464 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:35.001465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:35.001465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:35.001490 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:35.001701 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:35.001705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:35.001746 (+ 41us) client_negotiation.cc:769] Sending connection context
0501 02:07:35.001792 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 02:07:35.001887 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20240501 02:07:35.002071 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:34.995416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.22.12.126:39771)
0501 02:07:34.995472 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:34.995474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:34.995554 (+ 80us) server_negotiation.cc:407] Connection header received
0501 02:07:34.995600 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:34.995601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:34.995612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:34.995679 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:34.996301 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.997037 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:34.997546 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:34.997717 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:34.999479 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:34.999484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:34.999485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:34.999491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:35.001031 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:35.001387 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:35.001388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:35.001389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:35.001412 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:35.001537 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:35.001538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:35.001538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:35.001651 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:35.001713 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.001831 (+ 118us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.001950 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":30}
I20240501 02:07:35.004065 23689 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:56340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240501 02:07:35.004276 23689 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:35.005717 23826 tablet_service.cc:1467] Processing CreateTablet for tablet d7f29fe161454bd385f0012a6df4920d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b033d948d9a84ebd81a6012b49073505]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:07:35.005759 23966 tablet_service.cc:1467] Processing CreateTablet for tablet d7f29fe161454bd385f0012a6df4920d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b033d948d9a84ebd81a6012b49073505]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:07:35.005759 24106 tablet_service.cc:1467] Processing CreateTablet for tablet d7f29fe161454bd385f0012a6df4920d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b033d948d9a84ebd81a6012b49073505]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:07:35.005862 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f29fe161454bd385f0012a6df4920d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:35.005885 24106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f29fe161454bd385f0012a6df4920d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:35.005885 23966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f29fe161454bd385f0012a6df4920d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:35.007320 24196 tablet_bootstrap.cc:492] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: Bootstrap starting.
I20240501 02:07:35.007320 24195 tablet_bootstrap.cc:492] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: Bootstrap starting.
I20240501 02:07:35.007359 24199 tablet_bootstrap.cc:492] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: Bootstrap starting.
I20240501 02:07:35.007712 24195 tablet_bootstrap.cc:654] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:35.007712 24196 tablet_bootstrap.cc:654] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:35.007881 24199 tablet_bootstrap.cc:654] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:35.008596 24199 tablet_bootstrap.cc:492] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: No bootstrap required, opened a new log
I20240501 02:07:35.008652 24199 ts_tablet_manager.cc:1367] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:35.008828 24196 tablet_bootstrap.cc:492] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: No bootstrap required, opened a new log
I20240501 02:07:35.008801 24199 raft_consensus.cc:348] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.008864 24199 raft_consensus.cc:374] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:35.008883 24199 raft_consensus.cc:729] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52391792cc8d4cc6ad9da16f00484a57, State: Initialized, Role: FOLLOWER
I20240501 02:07:35.008875 24196 ts_tablet_manager.cc:1367] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:35.008924 24199 consensus_queue.cc:260] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.009024 24199 ts_tablet_manager.cc:1398] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:35.009016 24196 raft_consensus.cc:348] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.009097 24196 raft_consensus.cc:374] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:35.009124 24196 raft_consensus.cc:729] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3422e6ff77464cd0aa0b1daa30ae37ce, State: Initialized, Role: FOLLOWER
I20240501 02:07:35.009168 24196 consensus_queue.cc:260] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.009159 24203 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.009217 24203 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.009255 24196 ts_tablet_manager.cc:1398] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:35.009418 24200 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.009500 24200 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.009764 24195 tablet_bootstrap.cc:492] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: No bootstrap required, opened a new log
I20240501 02:07:35.009814 24195 ts_tablet_manager.cc:1367] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 02:07:35.009950 24195 raft_consensus.cc:348] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.010008 24195 raft_consensus.cc:374] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:35.010025 24195 raft_consensus.cc:729] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fa33e914c54ba1bdbc6d45695ac64b, State: Initialized, Role: FOLLOWER
I20240501 02:07:35.010066 24195 consensus_queue.cc:260] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.010175 24195 ts_tablet_manager.cc:1398] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:35.010192 24201 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.010282 24201 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.125208 24203 raft_consensus.cc:482] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:35.125320 24203 raft_consensus.cc:504] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.125639 24203 leader_election.cc:288] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787), 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483)
I20240501 02:07:35.128394 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.125997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:36483 (local address 127.22.12.65:33361)
0501 02:07:35.126149 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:35.126157 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:35.126208 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:35.126295 (+ 87us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:35.126296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:35.126297 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:35.126509 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.126510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.127493 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.127494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:35.128163 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.128165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.128224 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.128230 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:07:35.128262 (+ 32us) client_negotiation.cc:240] Negotiation successful
0501 02:07:35.128300 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20240501 02:07:35.128504 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.125792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:35787 (local address 127.22.12.65:60951)
0501 02:07:35.125965 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:35.125976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:35.126045 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:35.126201 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:35.126201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:35.126204 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:35.126482 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.126484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.127470 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.127472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:35.128215 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.128218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.128286 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.128290 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:35.128317 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:07:35.128354 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20240501 02:07:35.128811 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.126065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:33361 (local address 127.22.12.67:36483)
0501 02:07:35.126114 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.126116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.126213 (+ 97us) server_negotiation.cc:407] Connection header received
0501 02:07:35.126244 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.126245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.126253 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.126293 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:35.126595 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.127368 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.128248 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.128697 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.128708 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.128733 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.128762 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:07:35.128846 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.125856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:60951 (local address 127.22.12.66:35787)
0501 02:07:35.125908 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.125910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.126039 (+ 129us) server_negotiation.cc:407] Connection header received
0501 02:07:35.126105 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.126105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.126116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.126174 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:35.126584 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.127361 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.128291 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.128753 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.128760 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.128780 (+ 20us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.128803 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240501 02:07:35.129032 23986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f29fe161454bd385f0012a6df4920d" candidate_uuid: "52391792cc8d4cc6ad9da16f00484a57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" is_pre_election: true
I20240501 02:07:35.129117 23986 raft_consensus.cc:2454] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52391792cc8d4cc6ad9da16f00484a57 in term 0.
I20240501 02:07:35.129148 24126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f29fe161454bd385f0012a6df4920d" candidate_uuid: "52391792cc8d4cc6ad9da16f00484a57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" is_pre_election: true
I20240501 02:07:35.129235 24126 raft_consensus.cc:2454] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52391792cc8d4cc6ad9da16f00484a57 in term 0.
I20240501 02:07:35.129284 23779 leader_election.cc:302] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3422e6ff77464cd0aa0b1daa30ae37ce, 52391792cc8d4cc6ad9da16f00484a57; no voters:
I20240501 02:07:35.129419 24203 raft_consensus.cc:2781] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:35.129472 24203 raft_consensus.cc:482] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:35.129499 24203 raft_consensus.cc:3037] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:35.130101 24203 raft_consensus.cc:504] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.130218 24203 leader_election.cc:288] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [CANDIDATE]: Term 1 election: Requested vote from peers 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787), 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483)
I20240501 02:07:35.130411 23986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f29fe161454bd385f0012a6df4920d" candidate_uuid: "52391792cc8d4cc6ad9da16f00484a57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce"
I20240501 02:07:35.130424 24126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f29fe161454bd385f0012a6df4920d" candidate_uuid: "52391792cc8d4cc6ad9da16f00484a57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fa33e914c54ba1bdbc6d45695ac64b"
I20240501 02:07:35.130478 24126 raft_consensus.cc:3037] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:35.130481 23986 raft_consensus.cc:3037] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:35.131074 24126 raft_consensus.cc:2454] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52391792cc8d4cc6ad9da16f00484a57 in term 1.
I20240501 02:07:35.131098 23986 raft_consensus.cc:2454] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52391792cc8d4cc6ad9da16f00484a57 in term 1.
I20240501 02:07:35.131246 23779 leader_election.cc:302] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fa33e914c54ba1bdbc6d45695ac64b, 52391792cc8d4cc6ad9da16f00484a57; no voters:
I20240501 02:07:35.131354 24203 raft_consensus.cc:2781] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:35.131498 24203 raft_consensus.cc:686] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [term 1 LEADER]: Becoming Leader. State: Replica: 52391792cc8d4cc6ad9da16f00484a57, State: Running, Role: LEADER
I20240501 02:07:35.131563 24203 consensus_queue.cc:237] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } }
I20240501 02:07:35.131891 24238 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: TxnStatusTablet state changed. Reason: New leader 52391792cc8d4cc6ad9da16f00484a57. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.131983 24238 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:07:35.132151 24240 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240501 02:07:35.132232 24240 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240501 02:07:35.132261 23689 catalog_manager.cc:5484] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } health_report { overall_health: UNKNOWN } } }
I20240501 02:07:35.158409 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.156259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.22.12.65:35557)
0501 02:07:35.156307 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.156308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.156317 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:07:35.156434 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.156434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.156449 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.156508 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:35.156827 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.157304 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.157903 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.158050 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.158086 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.158167 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.158231 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.158303 (+ 72us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.158327 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:07:35.158429 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.156176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35557 (local address 127.0.0.1:59326)
0501 02:07:35.156261 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:35.156273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:35.156327 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:35.156526 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:35.156527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:35.156529 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:35.156764 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.156767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.157379 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.157380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:35.157820 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.157822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.157884 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.157915 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.158229 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.158239 (+ 10us) client_negotiation.cc:769] Sending connection context
0501 02:07:35.158296 (+ 57us) client_negotiation.cc:240] Negotiation successful
0501 02:07:35.158347 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20240501 02:07:35.159574 23986 raft_consensus.cc:1261] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Refusing update from remote peer 52391792cc8d4cc6ad9da16f00484a57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:35.159606 24126 raft_consensus.cc:1261] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Refusing update from remote peer 52391792cc8d4cc6ad9da16f00484a57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:35.159829 24238 consensus_queue.cc:1035] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:35.159997 24203 consensus_queue.cc:1035] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:35.160477 24200 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: TxnStatusTablet state changed. Reason: New leader 52391792cc8d4cc6ad9da16f00484a57. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.160575 24200 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.161031 24201 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: TxnStatusTablet state changed. Reason: New leader 52391792cc8d4cc6ad9da16f00484a57. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.161119 24201 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.161893 24200 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.161981 24200 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 3422e6ff77464cd0aa0b1daa30ae37ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.162102 24245 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.162191 24245 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:07:35.162271 24245 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.162317 24245 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 52391792cc8d4cc6ad9da16f00484a57: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:07:35.163167 24208 mvcc.cc:205] Tried to move back new op lower bound from 7022711829131694080 to 7022711829019422720. Current Snapshot: MvccSnapshot[applied={T|T < 7022711829131694080 or (T in {7022711829131694080})}]
I20240501 02:07:35.163379 24201 tablet_replica.cc:439] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52391792cc8d4cc6ad9da16f00484a57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 } } peers { permanent_uuid: "3422e6ff77464cd0aa0b1daa30ae37ce" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 35787 } } peers { permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 } } }
I20240501 02:07:35.163460 24201 tablet_replica.cc:442] T d7f29fe161454bd385f0012a6df4920d P 04fa33e914c54ba1bdbc6d45695ac64b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:35.166287 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.164402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.22.12.66:35787)
0501 02:07:35.164451 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.164453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.164460 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:07:35.164517 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.164518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.164536 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.164602 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:35.164803 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.165248 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.165836 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.165971 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.166006 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.166086 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.166148 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.166203 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.166222 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:07:35.171273 24248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.168651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35557 (local address 127.22.12.66:32887)
0501 02:07:35.168798 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:35.168808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:35.168866 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:35.169014 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:35.169015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:35.169017 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:35.169267 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.169268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.170180 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.170180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:35.170703 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.170705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.170770 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.170786 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.171085 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.171088 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:35.171142 (+ 54us) client_negotiation.cc:240] Negotiation successful
0501 02:07:35.171189 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20240501 02:07:35.171272 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.168743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:32887 (local address 127.22.12.65:35557)
0501 02:07:35.168790 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.168792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.168865 (+ 73us) server_negotiation.cc:407] Connection header received
0501 02:07:35.168910 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.168911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.168921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.168982 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:35.169336 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.170095 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.170775 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.170948 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.170969 (+ 21us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.170997 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.171058 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.171153 (+ 95us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.171176 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 02:07:35.173681 23846 raft_consensus.cc:1261] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Refusing update from remote peer 3422e6ff77464cd0aa0b1daa30ae37ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:35.173669 24126 raft_consensus.cc:1261] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Refusing update from remote peer 3422e6ff77464cd0aa0b1daa30ae37ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:35.173923 24200 consensus_queue.cc:1035] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "52391792cc8d4cc6ad9da16f00484a57" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:35.174067 24225 consensus_queue.cc:1035] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fa33e914c54ba1bdbc6d45695ac64b" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:35.183459 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.180905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:38333 (local address 127.22.12.66:35787)
0501 02:07:35.180952 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.180954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.181054 (+ 100us) server_negotiation.cc:407] Connection header received
0501 02:07:35.181122 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.181123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.181132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.181171 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:35.181497 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.182115 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.183067 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.183194 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.183211 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.183233 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.183287 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.183379 (+ 92us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.183404 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:07:35.183497 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.180794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:35787 (local address 127.22.12.65:38333)
0501 02:07:35.180989 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:35.180998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:35.181060 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:35.181182 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:35.181183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:35.181184 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:35.181433 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.181435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.182214 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.182215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:35.182977 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:35.182980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.183056 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.183078 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:35.183300 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:35.183302 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:07:35.183346 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 02:07:35.183392 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20240501 02:07:35.197962 23687 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:35.198089 23687 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:35.198858 23687 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d6534925d37347cf8458b1c41e1af593: Sending DeleteTablet for 3 replicas of tablet 302bedb9c0c5446bb816636bd47b1e24
I20240501 02:07:35.199213 23966 tablet_service.cc:1514] Processing DeleteTablet for tablet 302bedb9c0c5446bb816636bd47b1e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:35 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:46978
I20240501 02:07:35.199324 24219 tablet_replica.cc:330] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: stopping tablet replica
I20240501 02:07:35.199331 24106 tablet_service.cc:1514] Processing DeleteTablet for tablet 302bedb9c0c5446bb816636bd47b1e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:35 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:49048
I20240501 02:07:35.199379 24219 raft_consensus.cc:2229] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:35.199409 24218 tablet_replica.cc:330] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: stopping tablet replica
I20240501 02:07:35.199446 24218 raft_consensus.cc:2229] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:35.199479 24219 raft_consensus.cc:2258] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:35.199499 24218 raft_consensus.cc:2258] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:35.199810 23826 tablet_service.cc:1514] Processing DeleteTablet for tablet 302bedb9c0c5446bb816636bd47b1e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:35 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:59316
I20240501 02:07:35.199841 24218 ts_tablet_manager.cc:1875] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:35.199901 24217 tablet_replica.cc:330] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: stopping tablet replica
I20240501 02:07:35.199952 24217 raft_consensus.cc:2229] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:35.199986 24217 raft_consensus.cc:2258] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:35.200429 24217 ts_tablet_manager.cc:1875] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:35.202200 24217 ts_tablet_manager.cc:1888] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:35.202289 24217 log.cc:1196] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/302bedb9c0c5446bb816636bd47b1e24
I20240501 02:07:35.202489 24217 ts_tablet_manager.cc:1909] T 302bedb9c0c5446bb816636bd47b1e24 P 52391792cc8d4cc6ad9da16f00484a57: Deleting consensus metadata
I20240501 02:07:35.202797 24219 ts_tablet_manager.cc:1875] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:35.204294 24219 ts_tablet_manager.cc:1888] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:35.204360 24219 log.cc:1196] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/302bedb9c0c5446bb816636bd47b1e24
I20240501 02:07:35.204551 24219 ts_tablet_manager.cc:1909] T 302bedb9c0c5446bb816636bd47b1e24 P 3422e6ff77464cd0aa0b1daa30ae37ce: Deleting consensus metadata
I20240501 02:07:35.205456 24218 ts_tablet_manager.cc:1888] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:35.205521 24218 log.cc:1196] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/302bedb9c0c5446bb816636bd47b1e24
I20240501 02:07:35.205713 24218 ts_tablet_manager.cc:1909] T 302bedb9c0c5446bb816636bd47b1e24 P 04fa33e914c54ba1bdbc6d45695ac64b: Deleting consensus metadata
I20240501 02:07:35.206125 23675 catalog_manager.cc:4830] TS 04fa33e914c54ba1bdbc6d45695ac64b (127.22.12.67:36483): tablet 302bedb9c0c5446bb816636bd47b1e24 (table test-table [id=fad3e6eb3b3c4639a62116a6d13425ca]) successfully deleted
I20240501 02:07:35.206487 23675 catalog_manager.cc:4830] TS 3422e6ff77464cd0aa0b1daa30ae37ce (127.22.12.66:35787): tablet 302bedb9c0c5446bb816636bd47b1e24 (table test-table [id=fad3e6eb3b3c4639a62116a6d13425ca]) successfully deleted
I20240501 02:07:35.206890 23674 catalog_manager.cc:4830] TS 52391792cc8d4cc6ad9da16f00484a57 (127.22.12.65:35557): tablet 302bedb9c0c5446bb816636bd47b1e24 (table test-table [id=fad3e6eb3b3c4639a62116a6d13425ca]) successfully deleted
May 01 02:07:35 dist-test-slave-9tkh krb5kdc[23644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:35.217516 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.205603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.22.12.65:35557)
0501 02:07:35.205986 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.205989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.207088 (+ 1099us) server_negotiation.cc:407] Connection header received
0501 02:07:35.207186 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.207187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.207198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.207225 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:35.208198 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.208950 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.209760 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.209946 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.213508 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:35.213523 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:35.213523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:35.213532 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:35.216291 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:35.216683 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:35.216686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:35.216686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:35.216721 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:35.216868 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:35.216868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:35.216869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:35.216994 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:35.217099 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.217314 (+ 215us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.217413 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
W20240501 02:07:35.218150 23886 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59332
I20240501 02:07:35.229595 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:35.219118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.22.12.126:39771)
0501 02:07:35.219178 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:35.219180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:35.222748 (+ 3568us) server_negotiation.cc:407] Connection header received
0501 02:07:35.222827 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:35.222829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:35.222842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:35.222911 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:35.223723 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.224498 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:35.225277 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:35.225468 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:35.226772 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:35.226778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:35.226778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:35.226786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:35.228562 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:35.228908 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:35.228909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:35.228910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:35.228938 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:35.229089 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:35.229090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:35.229090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:35.229200 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:35.229303 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:35.229407 (+ 104us) server_negotiation.cc:299] Negotiation successful
0501 02:07:35.229494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20240501 02:07:35.230070 23687 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56342
I20240501 02:07:35.230703 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23761
I20240501 02:07:35.236699 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23901
I20240501 02:07:35.242091 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 24041
I20240501 02:07:35.247242 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 23657
2024-05-01T02:07:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2290 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:35 dist-test-slave-9tkh krb5kdc[24261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:35 dist-test-slave-9tkh krb5kdc[24261](info): set up 2 sockets
May 01 02:07:35 dist-test-slave-9tkh krb5kdc[24261](info): commencing operation
krb5kdc: starting...
W20240501 02:07:36.150095 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.883s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:36.263411 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:36261
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:36.344194 24274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:36.344408 24274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:36.344429 24274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:36.345731 24274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:36.345770 24274 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240501 02:07:36.345782 24274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:36.345793 24274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:36.345803 24274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:36.345813 24274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:36.347548 24274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:36261
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:36.347750 24274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:36.347904 24274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:36.352454 24282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.352751 24280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:36.353266 24274 server_base.cc:1034] running on GCE node
W20240501 02:07:36.353657 24279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:36.353952 24274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:36.354193 24274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:36.355333 24274 hybrid_clock.cc:648] HybridClock initialized: now 1714529256355314 us; error 35 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:36.369524 24274 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:36.370018 24274 webserver.cc:460] Webserver started at http://127.22.12.126:37867/ using document root <none> and password file <none>
I20240501 02:07:36.370211 24274 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:36.370254 24274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:36.370362 24274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:36.371412 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "6925a926955d4c7e8194a666cff54325"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "f34d9004486b01e4e82c55de847f4b1c"
server_key_iv: "6c979b47d1091c8f1291dca435e68fc1"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.371724 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "6925a926955d4c7e8194a666cff54325"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "f34d9004486b01e4e82c55de847f4b1c"
server_key_iv: "6c979b47d1091c8f1291dca435e68fc1"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.372946 24274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:36.373704 24289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.373884 24274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:36.373970 24274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "6925a926955d4c7e8194a666cff54325"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "f34d9004486b01e4e82c55de847f4b1c"
server_key_iv: "6c979b47d1091c8f1291dca435e68fc1"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.374029 24274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:36.384826 24274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:36.386821 24274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:36.386940 24274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:36.391286 24274 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:36261
I20240501 02:07:36.391328 24351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:36261 every 8 connection(s)
I20240501 02:07:36.391644 24274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:36.392200 24352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:36.394277 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Bootstrap starting.
I20240501 02:07:36.394873 24352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:36.395110 24352 log.cc:824] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:36.395738 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: No bootstrap required, opened a new log
I20240501 02:07:36.397509 24352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925a926955d4c7e8194a666cff54325" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36261 } }
I20240501 02:07:36.397634 24352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:36.397650 24352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6925a926955d4c7e8194a666cff54325, State: Initialized, Role: FOLLOWER
I20240501 02:07:36.397764 24352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925a926955d4c7e8194a666cff54325" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36261 } }
I20240501 02:07:36.397828 24352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:36.397850 24352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:36.397894 24352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:36.398093 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 24274
I20240501 02:07:36.398192 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:36.398345 22577 external_mini_cluster.cc:1428] Setting key d967ba2e62412bcec2067ff4ae556136
I20240501 02:07:36.398473 24352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925a926955d4c7e8194a666cff54325" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36261 } }
I20240501 02:07:36.398595 24352 leader_election.cc:302] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6925a926955d4c7e8194a666cff54325; no voters:
I20240501 02:07:36.398762 24352 leader_election.cc:288] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:36.398933 24357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:36.399001 24352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:36.399075 24357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [term 1 LEADER]: Becoming Leader. State: Replica: 6925a926955d4c7e8194a666cff54325, State: Running, Role: LEADER
I20240501 02:07:36.399164 24357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925a926955d4c7e8194a666cff54325" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36261 } }
I20240501 02:07:36.399475 24357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6925a926955d4c7e8194a666cff54325. Latest consensus state: current_term: 1 leader_uuid: "6925a926955d4c7e8194a666cff54325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925a926955d4c7e8194a666cff54325" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36261 } } }
I20240501 02:07:36.399564 24357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:36.399492 24355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6925a926955d4c7e8194a666cff54325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6925a926955d4c7e8194a666cff54325" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36261 } } }
I20240501 02:07:36.399737 24355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:36.399755 24366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:36.400338 24366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:36.402117 24366 catalog_manager.cc:1347] Generated new cluster ID: 2c97ea142e9f448fb8f71a3020f080e0
I20240501 02:07:36.402173 24366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:36.409081 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.398818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.22.12.126:36261)
0501 02:07:36.399073 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.399088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.399190 (+ 102us) server_negotiation.cc:407] Connection header received
0501 02:07:36.399450 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.399458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.399509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.399663 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.400713 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.401674 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.402410 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.403117 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.405106 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.405112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.405113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.405133 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.407677 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.408070 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.408073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.408073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.408129 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.408274 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.408274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.408274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.408493 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.408628 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.408749 (+ 121us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.408869 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20240501 02:07:36.419718 24366 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:36.420399 24366 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:36.426954 24366 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Generated new TSK 0
I20240501 02:07:36.427181 24366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:36.453735 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36261
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:36.533957 24378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:36.534158 24378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:36.534183 24378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:36.535606 24378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:36.535657 24378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:36.535677 24378 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240501 02:07:36.535687 24378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:07:36.535703 24378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:36.537299 24378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:36.537515 24378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:36.537725 24378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:36.542100 24386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.542485 24384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.542901 24383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:36.543315 24378 server_base.cc:1034] running on GCE node
I20240501 02:07:36.543510 24378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:36.543745 24378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:36.544873 24378 hybrid_clock.cc:648] HybridClock initialized: now 1714529256544852 us; error 41 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:36.547263 24378 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:36.547684 24378 webserver.cc:460] Webserver started at http://127.22.12.65:43797/ using document root <none> and password file <none>
I20240501 02:07:36.547876 24378 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:36.547914 24378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:36.548013 24378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:36.549551 24378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "f02930bf38144b73b5aa56d16428771e"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "b950975994dcbfd6a9282502daf3daf5"
server_key_iv: "3b09be1ea5da64b6cdb279ec7c2546be"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.549860 24378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "f02930bf38144b73b5aa56d16428771e"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "b950975994dcbfd6a9282502daf3daf5"
server_key_iv: "3b09be1ea5da64b6cdb279ec7c2546be"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.551106 24378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:36.551740 24393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.551954 24378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:36.552024 24378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "f02930bf38144b73b5aa56d16428771e"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "b950975994dcbfd6a9282502daf3daf5"
server_key_iv: "3b09be1ea5da64b6cdb279ec7c2546be"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.552076 24378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:36.563838 24378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:36.565363 24378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:36.565454 24378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:36.565879 24378 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:36.565922 24378 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.565943 24378 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:36.565954 24378 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:36.571995 24378 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38073
I20240501 02:07:36.572258 24508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38073 every 8 connection(s)
I20240501 02:07:36.572338 24378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:36.574438 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.566298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:39211 (local address 127.22.12.126:36261)
0501 02:07:36.566368 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.566371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.566495 (+ 124us) server_negotiation.cc:407] Connection header received
0501 02:07:36.567018 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.567019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.567032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.567111 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.568101 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.568863 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.569599 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.569774 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.571997 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.572002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.572002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.572010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.573419 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.573750 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.573751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.573751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.573768 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.573954 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.573954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.573955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.574041 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.574096 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.574297 (+ 201us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.574370 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:07:36.574563 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.566255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.22.12.65:39211)
0501 02:07:36.566390 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.566418 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.566921 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.567159 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.567162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.567533 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:36.567992 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.567999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.568960 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.568961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.569511 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.569514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.569664 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.570264 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:36.570290 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:36.571907 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.573490 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.573495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.573498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.573691 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.573829 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.573830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.573830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.573886 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.574094 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.574097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:36.574202 (+ 105us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.574284 (+ 82us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.574389 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20240501 02:07:36.575013 24509 heartbeater.cc:347] Connected to a master server at 127.22.12.126:36261
I20240501 02:07:36.575096 24509 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:36.575284 24509 heartbeater.cc:511] Master 127.22.12.126:36261 requested a full tablet report, sending...
I20240501 02:07:36.575819 24305 ts_manager.cc:194] Registered new tserver with Master: f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073)
I20240501 02:07:36.576829 24305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:39211
I20240501 02:07:36.578234 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 24378
I20240501 02:07:36.578353 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:36.578567 22577 external_mini_cluster.cc:1428] Setting key 937abd73bef695fc83020f28f0d9f0df
I20240501 02:07:36.580817 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.575876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.22.12.65:37627)
0501 02:07:36.576017 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.576028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.576114 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.576235 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.576236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.576422 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:36.576676 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.576678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.577330 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.577331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.577709 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.577710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.577768 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.578198 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:36.578202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:36.578509 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.580033 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.580034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.580035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.580201 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.580330 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.580330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.580331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.580357 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.580515 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.580515 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:36.580558 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.580623 (+ 65us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.580722 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20240501 02:07:36.580818 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.575974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:37627 (local address 127.22.12.126:36261)
0501 02:07:36.576025 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.576027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.576109 (+ 82us) server_negotiation.cc:407] Connection header received
0501 02:07:36.576168 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.576169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.576177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.576219 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.576742 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.577268 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.577777 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.577942 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.578599 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.578603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.578603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.578610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.579959 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.580267 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.580268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.580268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.580285 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.580399 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.580399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.580399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.580472 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.580513 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.580653 (+ 140us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.580732 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:36.605182 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36261
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:36.683284 24518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:36.683487 24518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:36.683513 24518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:36.684862 24518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:36.684912 24518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:36.684932 24518 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240501 02:07:36.684942 24518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:07:36.684958 24518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:36.686580 24518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:36.686808 24518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:36.686998 24518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:36.691474 24526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.691681 24524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.692071 24523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:36.692493 24518 server_base.cc:1034] running on GCE node
I20240501 02:07:36.692678 24518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:36.692873 24518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:36.693996 24518 hybrid_clock.cc:648] HybridClock initialized: now 1714529256693981 us; error 34 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:36.696336 24518 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:36.696745 24518 webserver.cc:460] Webserver started at http://127.22.12.66:35203/ using document root <none> and password file <none>
I20240501 02:07:36.696907 24518 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:36.696949 24518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:36.697054 24518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:36.698037 24518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "8ba839eb732d40d198674ef83297c008"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "f41c85d380d1ace834160d504335a789"
server_key_iv: "a0c04359fe9781b90f0c21ef2fa61372"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.698324 24518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "8ba839eb732d40d198674ef83297c008"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "f41c85d380d1ace834160d504335a789"
server_key_iv: "a0c04359fe9781b90f0c21ef2fa61372"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.699576 24518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:36.700286 24533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.700420 24518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:36.700503 24518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "8ba839eb732d40d198674ef83297c008"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "f41c85d380d1ace834160d504335a789"
server_key_iv: "a0c04359fe9781b90f0c21ef2fa61372"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.700556 24518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:36.733337 24518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:36.735342 24518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:36.735452 24518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:36.735940 24518 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:36.735987 24518 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.736016 24518 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:36.736032 24518 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.741801 24518 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:43193
I20240501 02:07:36.741850 24648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:43193 every 8 connection(s)
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:36.742154 24518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:36.744935 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.736360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:60027 (local address 127.22.12.126:36261)
0501 02:07:36.736425 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.736427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.736609 (+ 182us) server_negotiation.cc:407] Connection header received
0501 02:07:36.737193 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.737195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.737208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.737296 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.738348 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.739139 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.739871 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.740060 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.742474 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.742479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.742480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.742488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.743854 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.744227 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.744228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.744228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.744245 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.744384 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.744384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.744384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.744472 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.744516 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.744799 (+ 283us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.744866 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:07:36.745098 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.736313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.22.12.66:60027)
0501 02:07:36.736487 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.736516 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.737067 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.737362 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.737367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.737810 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:36.738233 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.738240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.739238 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.739239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.739796 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.739799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.739928 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.740868 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:36.740883 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:36.742361 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.743954 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.743959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.743962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.744162 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.744291 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.744291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.744292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.744347 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.744556 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.744561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:36.744684 (+ 123us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.744787 (+ 103us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.744928 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20240501 02:07:36.745570 24649 heartbeater.cc:347] Connected to a master server at 127.22.12.126:36261
I20240501 02:07:36.745656 24649 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:36.745875 24649 heartbeater.cc:511] Master 127.22.12.126:36261 requested a full tablet report, sending...
I20240501 02:07:36.746294 24306 ts_manager.cc:194] Registered new tserver with Master: 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193)
I20240501 02:07:36.746796 24306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:60027
I20240501 02:07:36.750010 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 24518
I20240501 02:07:36.750118 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:36.750289 22577 external_mini_cluster.cc:1428] Setting key de36aff9aafb86c21e3c277a691f8da3
I20240501 02:07:36.751483 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.746590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.22.12.66:57569)
0501 02:07:36.746725 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.746735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.746802 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.746973 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.746975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.747118 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:36.747347 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.747350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.747981 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.747982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.748325 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.748327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.748381 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.748816 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:36.748820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:36.749040 (+ 220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.750595 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.750596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.750596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.750764 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.750968 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.750968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.750969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.751001 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.751257 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.751257 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:36.751292 (+ 35us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.751336 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.751408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20240501 02:07:36.751526 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.746621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:57569 (local address 127.22.12.126:36261)
0501 02:07:36.746664 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.746665 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.746798 (+ 133us) server_negotiation.cc:407] Connection header received
0501 02:07:36.746859 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.746860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.746870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.746922 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.747441 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.747905 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.748387 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.748504 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.749130 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.749134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.749134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.749140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.750488 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.750854 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.750856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.750856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.750887 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.751054 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.751054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.751054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.751195 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.751261 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.751372 (+ 111us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.751445 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:36.775889 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36261
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:36.853801 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:36.853996 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:36.854030 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:36.855417 24658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:36.855468 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:36.855492 24658 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240501 02:07:36.855506 24658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:07:36.855530 24658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:36.857124 24658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:36.857352 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:36.857537 24658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:36.861760 24663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.862098 24666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:36.862644 24664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:36.863003 24658 server_base.cc:1034] running on GCE node
I20240501 02:07:36.863166 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:36.863363 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:36.864487 24658 hybrid_clock.cc:648] HybridClock initialized: now 1714529256864469 us; error 34 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:36.866816 24658 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:36.867257 24658 webserver.cc:460] Webserver started at http://127.22.12.67:41053/ using document root <none> and password file <none>
I20240501 02:07:36.867448 24658 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:36.867489 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:36.867590 24658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:36.868530 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "4953e54ea493b0f9b7c6d4d0eaa12702"
server_key_iv: "d933b6bcbc3691da92b6f9623290e67f"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.868834 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "4953e54ea493b0f9b7c6d4d0eaa12702"
server_key_iv: "d933b6bcbc3691da92b6f9623290e67f"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.870020 24658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:07:36.870687 24673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.870832 24658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:36.870905 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
format_stamp: "Formatted at 2024-05-01 02:07:36 on dist-test-slave-9tkh"
server_key: "4953e54ea493b0f9b7c6d4d0eaa12702"
server_key_iv: "d933b6bcbc3691da92b6f9623290e67f"
server_key_version: "encryptionkey@0"
I20240501 02:07:36.870985 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:36.885164 24658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:36.886886 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:36.887001 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:36.887517 24658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:36.887558 24658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:36.887588 24658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:36.887617 24658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:36.894484 24658 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:33757
I20240501 02:07:36.894559 24788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:33757 every 8 connection(s)
I20240501 02:07:36.894891 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:36.896027 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.887868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:54771 (local address 127.22.12.126:36261)
0501 02:07:36.887933 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.887936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.888097 (+ 161us) server_negotiation.cc:407] Connection header received
0501 02:07:36.888586 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.888588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.888601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.888677 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.889418 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.890170 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.890819 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.890986 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.893224 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.893228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.893229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.893235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.894901 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.895303 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.895305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.895305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.895334 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.895514 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.895514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.895514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.895603 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.895649 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.895895 (+ 246us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.895959 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 02:07:36.896133 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.887833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.22.12.67:54771)
0501 02:07:36.887962 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.887990 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.888488 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.888728 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.888732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.889016 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:36.889336 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.889340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.890286 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.890288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.890729 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.890732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.890873 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.891499 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:36.891528 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:36.893128 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.894983 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.894986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.894989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.895213 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.895394 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.895395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.895396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.895452 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.895668 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.895673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:36.895788 (+ 115us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.895879 (+ 91us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.895989 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20240501 02:07:36.896576 24789 heartbeater.cc:347] Connected to a master server at 127.22.12.126:36261
I20240501 02:07:36.896653 24789 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:36.896835 24789 heartbeater.cc:511] Master 127.22.12.126:36261 requested a full tablet report, sending...
I20240501 02:07:36.897198 24306 ts_manager.cc:194] Registered new tserver with Master: b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:36.897615 24306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:54771
I20240501 02:07:36.900328 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 24658
I20240501 02:07:36.900416 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:36.900589 22577 external_mini_cluster.cc:1428] Setting key 6379cf648eb99ad39decfefac08b0d28
I20240501 02:07:36.901487 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:07:36.902298 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.897577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:54907 (local address 127.22.12.126:36261)
0501 02:07:36.897615 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.897616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.897728 (+ 112us) server_negotiation.cc:407] Connection header received
0501 02:07:36.897826 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.897826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.897834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.897884 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.898282 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.898784 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.899273 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.899749 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.900050 (+ 301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.900053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.900054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.900058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.901504 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.901743 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.901744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.901745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.901766 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.901889 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.901890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.901890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.901968 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.902024 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.902141 (+ 117us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.902206 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:07:36.902312 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.897514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.22.12.67:54907)
0501 02:07:36.897632 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.897642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.897707 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.897914 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.897915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.898047 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:36.898214 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.898215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.898851 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.898852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.899207 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.899208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.899264 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.899739 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:36.899743 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:36.899980 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.901566 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.901567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.901568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.901689 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.901813 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.901813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:36.901814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:36.901840 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.902016 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.902017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:36.902055 (+ 38us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.902114 (+ 59us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.902206 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:07:36 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:36.921444 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.915913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.22.12.126:36261)
0501 02:07:36.915969 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.915971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.915980 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:07:36.916005 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.916006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.916014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.916063 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:36.916687 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.917176 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.917669 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.917774 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.919636 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.919641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:36.919641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:36.919648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:36.920753 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.920992 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.920993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.920993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.921014 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.921102 (+ 88us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.921102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:36.921103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:36.921184 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.921252 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.921317 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.921373 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:07:36.923142 24306 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:36.924551 24306 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:36.929143 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.926411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33757 (local address 127.0.0.1:57044)
0501 02:07:36.926670 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.926677 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.926718 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.927102 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.927103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.927104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:36.927324 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.927327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928254 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.928255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.928846 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.928848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928917 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.928964 (+ 47us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.929012 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.929054 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":77,"threads_started":1}
I20240501 02:07:36.929831 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.926406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38073 (local address 127.0.0.1:37588)
0501 02:07:36.926617 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.926642 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.926713 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.927022 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.927024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.927026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:36.927545 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.927546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928834 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.928835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.929632 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.929635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.929694 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.929697 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.929725 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.929764 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":64,"threads_started":1}
I20240501 02:07:36.929957 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.926755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.22.12.67:33757)
0501 02:07:36.926903 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.926910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.926917 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:07:36.926945 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.926947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.926958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.927027 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:36.927405 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928177 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.928946 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.929767 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.929822 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.929854 (+ 32us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.929893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20240501 02:07:36.930481 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.926675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.22.12.66:43193)
0501 02:07:36.927114 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.927120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.927126 (+ 6us) server_negotiation.cc:407] Connection header received
0501 02:07:36.927152 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.927153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.927163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.927227 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:36.927532 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928730 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.929882 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.930350 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.930382 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.930408 (+ 26us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.930434 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":73,"threads_started":1}
I20240501 02:07:36.930624 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.926407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43193 (local address 127.0.0.1:32850)
0501 02:07:36.926892 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.926902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:36.926958 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.927241 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:36.927242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:36.927244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:36.927452 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.927454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928826 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.928827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:36.929683 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:36.929685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.929746 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.929750 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:36.929775 (+ 25us) client_negotiation.cc:240] Negotiation successful
0501 02:07:36.929811 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":7,"thread_start_us":63,"threads_started":1}
I20240501 02:07:36.930644 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:36.926604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.22.12.65:38073)
0501 02:07:36.926840 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:36.926848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:36.926859 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:07:36.926899 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:36.926902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:36.926920 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.927007 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:36.927603 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.928757 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.929736 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.930423 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.930467 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:36.930500 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:07:36.930576 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20240501 02:07:36.930589 24709 tablet_service.cc:1467] Processing CreateTablet for tablet e49079f49c3545459f3e6dbea6f3b7cd (DEFAULT_TABLE table=test-table [id=4e3a9bf8f46040b58327378dda901f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:36.930908 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49079f49c3545459f3e6dbea6f3b7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:36.931143 24443 tablet_service.cc:1467] Processing CreateTablet for tablet e49079f49c3545459f3e6dbea6f3b7cd (DEFAULT_TABLE table=test-table [id=4e3a9bf8f46040b58327378dda901f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:36.931149 24583 tablet_service.cc:1467] Processing CreateTablet for tablet e49079f49c3545459f3e6dbea6f3b7cd (DEFAULT_TABLE table=test-table [id=4e3a9bf8f46040b58327378dda901f35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:36.931393 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49079f49c3545459f3e6dbea6f3b7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:36.931393 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49079f49c3545459f3e6dbea6f3b7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:36.933073 24809 tablet_bootstrap.cc:492] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Bootstrap starting.
I20240501 02:07:36.933858 24809 tablet_bootstrap.cc:654] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:36.934144 24810 tablet_bootstrap.cc:492] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Bootstrap starting.
I20240501 02:07:36.934163 24809 log.cc:824] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:36.934423 24811 tablet_bootstrap.cc:492] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Bootstrap starting.
I20240501 02:07:36.935019 24809 tablet_bootstrap.cc:492] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: No bootstrap required, opened a new log
I20240501 02:07:36.935094 24810 tablet_bootstrap.cc:654] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:36.935091 24809 ts_tablet_manager.cc:1367] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:36.935242 24811 tablet_bootstrap.cc:654] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:36.935369 24810 log.cc:824] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:36.935488 24811 log.cc:824] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:36.936021 24810 tablet_bootstrap.cc:492] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: No bootstrap required, opened a new log
I20240501 02:07:36.936075 24810 ts_tablet_manager.cc:1367] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:36.936133 24811 tablet_bootstrap.cc:492] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: No bootstrap required, opened a new log
I20240501 02:07:36.936203 24811 ts_tablet_manager.cc:1367] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:36.936830 24809 raft_consensus.cc:348] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:36.936976 24809 raft_consensus.cc:374] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:36.937001 24809 raft_consensus.cc:729] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67fd7cc79f44c1d83a054d773f05d4e, State: Initialized, Role: FOLLOWER
I20240501 02:07:36.937094 24809 consensus_queue.cc:260] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:36.937338 24809 ts_tablet_manager.cc:1398] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 02:07:36.937436 24789 heartbeater.cc:503] Master 127.22.12.126:36261 was elected leader, sending a full tablet report...
I20240501 02:07:36.937427 24810 raft_consensus.cc:348] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:36.937577 24810 raft_consensus.cc:374] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:36.937608 24810 raft_consensus.cc:729] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02930bf38144b73b5aa56d16428771e, State: Initialized, Role: FOLLOWER
I20240501 02:07:36.937680 24810 consensus_queue.cc:260] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:36.937906 24810 ts_tablet_manager.cc:1398] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:07:36.937966 24509 heartbeater.cc:503] Master 127.22.12.126:36261 was elected leader, sending a full tablet report...
I20240501 02:07:36.938242 24811 raft_consensus.cc:348] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:36.938381 24811 raft_consensus.cc:374] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:36.938575 24811 raft_consensus.cc:729] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Initialized, Role: FOLLOWER
I20240501 02:07:36.938676 24811 consensus_queue.cc:260] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:36.938920 24811 ts_tablet_manager.cc:1398] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 02:07:36.938948 24649 heartbeater.cc:503] Master 127.22.12.126:36261 was elected leader, sending a full tablet report...
W20240501 02:07:36.992677 24650 tablet.cc:2369] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:37.053134 24817 raft_consensus.cc:482] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:37.053257 24817 raft_consensus.cc:504] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:37.053606 24817 leader_election.cc:288] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:37.055902 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.053786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38073 (local address 127.22.12.66:47999)
0501 02:07:37.053846 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.053857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.053900 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.054053 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.054054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.054056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:37.054276 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.054278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.055046 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.055047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.055656 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.055658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.055724 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.055727 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.055753 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.055786 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240501 02:07:37.055915 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.053840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33757 (local address 127.22.12.66:44677)
0501 02:07:37.054008 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.054016 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.054066 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.054187 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.054188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.054189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:37.054417 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.054418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.055066 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.055066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.055651 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.055653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.055715 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.055721 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.055749 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.055785 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20240501 02:07:37.056248 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.053843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:47999 (local address 127.22.12.65:38073)
0501 02:07:37.053880 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.053881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.053891 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:37.053956 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.053957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.053967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.054010 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:37.054370 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.054963 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.055733 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.056138 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.056145 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.056165 (+ 20us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.056193 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:07:37.056281 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.053951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:44677 (local address 127.22.12.67:33757)
0501 02:07:37.054008 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.054010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.054073 (+ 63us) server_negotiation.cc:407] Connection header received
0501 02:07:37.054103 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.054104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.054114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.054183 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:37.054486 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.054983 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.055726 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.056160 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.056169 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.056191 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.056223 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:07:37.056533 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49079f49c3545459f3e6dbea6f3b7cd" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e" is_pre_election: true
I20240501 02:07:37.056543 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49079f49c3545459f3e6dbea6f3b7cd" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" is_pre_election: true
I20240501 02:07:37.056669 24461 raft_consensus.cc:2454] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 0.
I20240501 02:07:37.056676 24743 raft_consensus.cc:2454] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 0.
I20240501 02:07:37.056867 24534 leader_election.cc:302] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:37.056982 24817 raft_consensus.cc:2781] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:37.057035 24817 raft_consensus.cc:482] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:37.057051 24817 raft_consensus.cc:3037] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.057689 24817 raft_consensus.cc:504] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:37.057814 24817 leader_election.cc:288] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 election: Requested vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:37.057960 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49079f49c3545459f3e6dbea6f3b7cd" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e"
I20240501 02:07:37.058032 24461 raft_consensus.cc:3037] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.058033 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49079f49c3545459f3e6dbea6f3b7cd" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
I20240501 02:07:37.058115 24743 raft_consensus.cc:3037] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.058730 24461 raft_consensus.cc:2454] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 1.
I20240501 02:07:37.058808 24743 raft_consensus.cc:2454] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 1.
I20240501 02:07:37.058964 24534 leader_election.cc:302] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:37.059073 24817 raft_consensus.cc:2781] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:37.059212 24817 raft_consensus.cc:686] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Running, Role: LEADER
I20240501 02:07:37.059289 24817 consensus_queue.cc:237] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } }
I20240501 02:07:37.060052 24305 catalog_manager.cc:5484] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba839eb732d40d198674ef83297c008 (127.22.12.66). New cstate: current_term: 1 leader_uuid: "8ba839eb732d40d198674ef83297c008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } health_report { overall_health: HEALTHY } } }
W20240501 02:07:37.073062 24510 tablet.cc:2369] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 01 02:07:37 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:37.105581 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.099372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36261 (local address 127.0.0.1:41010)
0501 02:07:37.099535 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.099544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.099605 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.099740 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.099741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.100135 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:37.100298 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.100300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.100904 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.100905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.101291 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.101293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.101343 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.101817 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:37.101821 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:37.103280 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:37.104821 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:37.104827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:37.104830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:37.105016 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:37.105131 (+ 115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:37.105132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:37.105132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:37.105156 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:37.105325 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:37.105328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:37.105368 (+ 40us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.105414 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.105505 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20240501 02:07:37.105654 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.099446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.22.12.126:36261)
0501 02:07:37.099503 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.099504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.099600 (+ 96us) server_negotiation.cc:407] Connection header received
0501 02:07:37.099653 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.099654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.099666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.099727 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:37.100364 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.100840 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.101348 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.101448 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.103357 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:37.103362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:37.103362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:37.103368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:37.104752 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:37.105072 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:37.105073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:37.105074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:37.105090 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:37.105197 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:37.105197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:37.105198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:37.105286 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:37.105329 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.105439 (+ 110us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.105557 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":34}
I20240501 02:07:37.107563 24305 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:41010:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240501 02:07:37.107774 24305 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:37.109148 24583 tablet_service.cc:1467] Processing CreateTablet for tablet ea3dbab4d9204257b626be95c79bc6ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7754711a70cc45cfa053117d53d9a9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:07:37.109133 24443 tablet_service.cc:1467] Processing CreateTablet for tablet ea3dbab4d9204257b626be95c79bc6ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7754711a70cc45cfa053117d53d9a9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:07:37.109292 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3dbab4d9204257b626be95c79bc6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:37.109292 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3dbab4d9204257b626be95c79bc6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:37.109257 24709 tablet_service.cc:1467] Processing CreateTablet for tablet ea3dbab4d9204257b626be95c79bc6ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7754711a70cc45cfa053117d53d9a9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:07:37.109395 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3dbab4d9204257b626be95c79bc6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:37.110574 24809 tablet_bootstrap.cc:492] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: Bootstrap starting.
I20240501 02:07:37.110736 24810 tablet_bootstrap.cc:492] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: Bootstrap starting.
I20240501 02:07:37.110952 24809 tablet_bootstrap.cc:654] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:37.111068 24810 tablet_bootstrap.cc:654] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:37.111460 24811 tablet_bootstrap.cc:492] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: Bootstrap starting.
I20240501 02:07:37.111740 24810 tablet_bootstrap.cc:492] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: No bootstrap required, opened a new log
I20240501 02:07:37.111791 24810 ts_tablet_manager.cc:1367] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:37.111845 24811 tablet_bootstrap.cc:654] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:37.111863 24809 tablet_bootstrap.cc:492] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: No bootstrap required, opened a new log
I20240501 02:07:37.111900 24809 ts_tablet_manager.cc:1367] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:37.111934 24810 raft_consensus.cc:348] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.112010 24810 raft_consensus.cc:374] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:37.112033 24810 raft_consensus.cc:729] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02930bf38144b73b5aa56d16428771e, State: Initialized, Role: FOLLOWER
I20240501 02:07:37.112049 24809 raft_consensus.cc:348] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.112071 24810 consensus_queue.cc:260] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.112113 24809 raft_consensus.cc:374] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:37.112138 24809 raft_consensus.cc:729] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67fd7cc79f44c1d83a054d773f05d4e, State: Initialized, Role: FOLLOWER
I20240501 02:07:37.112183 24810 ts_tablet_manager.cc:1398] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:37.112176 24809 consensus_queue.cc:260] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.112252 24809 ts_tablet_manager.cc:1398] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:37.112318 24816 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.112382 24816 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:37.112483 24815 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.112566 24815 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:37.113662 24811 tablet_bootstrap.cc:492] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: No bootstrap required, opened a new log
I20240501 02:07:37.113706 24811 ts_tablet_manager.cc:1367] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20240501 02:07:37.113829 24811 raft_consensus.cc:348] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.113883 24811 raft_consensus.cc:374] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:37.113895 24811 raft_consensus.cc:729] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Initialized, Role: FOLLOWER
I20240501 02:07:37.113940 24811 consensus_queue.cc:260] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.114027 24811 ts_tablet_manager.cc:1398] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:37.114075 24817 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.114151 24817 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: This TxnStatusTablet replica's current role is: FOLLOWER
W20240501 02:07:37.145538 24790 tablet.cc:2369] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:37.350900 24815 raft_consensus.cc:482] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:37.351002 24815 raft_consensus.cc:504] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.351327 24815 leader_election.cc:288] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193)
I20240501 02:07:37.353339 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.351512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38073 (local address 127.22.12.67:44827)
0501 02:07:37.351574 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.351584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.351630 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.351770 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.351771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.351773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:37.351946 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.351948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.352554 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.352555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.353133 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.353135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.353199 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.353203 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.353231 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.353265 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20240501 02:07:37.353438 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.351525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43193 (local address 127.22.12.67:52069)
0501 02:07:37.351679 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.351686 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.351743 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.351864 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.351865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.351867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:37.352052 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.352053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.352637 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.352638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.353211 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.353213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.353286 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.353291 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.353326 (+ 35us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.353360 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20240501 02:07:37.353791 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.351592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:44827 (local address 127.22.12.65:38073)
0501 02:07:37.351632 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.351634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.351644 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:37.351691 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.351692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.351701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.351746 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:37.352020 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.352475 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.353221 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.353672 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.353681 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.353702 (+ 21us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.353732 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:07:37.353973 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3dbab4d9204257b626be95c79bc6ca" candidate_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e" is_pre_election: true
I20240501 02:07:37.354051 24461 raft_consensus.cc:2454] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67fd7cc79f44c1d83a054d773f05d4e in term 0.
I20240501 02:07:37.354121 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.351586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:52069 (local address 127.22.12.66:43193)
0501 02:07:37.351639 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.351641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.351742 (+ 101us) server_negotiation.cc:407] Connection header received
0501 02:07:37.351802 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.351802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.351810 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.351855 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:37.352115 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.352569 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.353290 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.353998 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.354006 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.354032 (+ 26us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.354063 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:07:37.354337 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3dbab4d9204257b626be95c79bc6ca" candidate_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba839eb732d40d198674ef83297c008" is_pre_election: true
I20240501 02:07:37.354332 24674 leader_election.cc:302] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b67fd7cc79f44c1d83a054d773f05d4e, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:37.354447 24603 raft_consensus.cc:2454] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67fd7cc79f44c1d83a054d773f05d4e in term 0.
I20240501 02:07:37.354557 24815 raft_consensus.cc:2781] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:37.354612 24815 raft_consensus.cc:482] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:37.354650 24815 raft_consensus.cc:3037] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.355729 24815 raft_consensus.cc:504] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.355875 24815 leader_election.cc:288] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [CANDIDATE]: Term 1 election: Requested vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193)
I20240501 02:07:37.356068 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3dbab4d9204257b626be95c79bc6ca" candidate_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e"
I20240501 02:07:37.356135 24461 raft_consensus.cc:3037] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.356096 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3dbab4d9204257b626be95c79bc6ca" candidate_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba839eb732d40d198674ef83297c008"
I20240501 02:07:37.356185 24603 raft_consensus.cc:3037] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.356834 24461 raft_consensus.cc:2454] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67fd7cc79f44c1d83a054d773f05d4e in term 1.
I20240501 02:07:37.356971 24603 raft_consensus.cc:2454] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67fd7cc79f44c1d83a054d773f05d4e in term 1.
I20240501 02:07:37.356973 24674 leader_election.cc:302] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b67fd7cc79f44c1d83a054d773f05d4e, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:37.357153 24815 raft_consensus.cc:2781] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:37.357295 24815 raft_consensus.cc:686] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [term 1 LEADER]: Becoming Leader. State: Replica: b67fd7cc79f44c1d83a054d773f05d4e, State: Running, Role: LEADER
I20240501 02:07:37.357391 24815 consensus_queue.cc:237] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.357797 24831 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: TxnStatusTablet state changed. Reason: New leader b67fd7cc79f44c1d83a054d773f05d4e. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.357877 24831 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:07:37.358119 24833 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240501 02:07:37.358228 24833 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240501 02:07:37.358175 24306 catalog_manager.cc:5484] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e reported cstate change: term changed from 0 to 1, leader changed from <none> to b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67). New cstate: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } health_report { overall_health: HEALTHY } } }
I20240501 02:07:37.399127 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.396724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.22.12.67:33757)
0501 02:07:37.396760 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.396762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.396801 (+ 39us) server_negotiation.cc:407] Connection header received
0501 02:07:37.396904 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.396904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.396921 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.396982 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:37.397329 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.397771 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.398656 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.398777 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.398809 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.398904 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.398966 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.399031 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.399050 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:07:37.399155 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.396659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33757 (local address 127.0.0.1:57056)
0501 02:07:37.396756 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.396767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.396821 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.397007 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.397009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.397010 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:37.397265 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.397267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.397844 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.397845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.398582 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.398585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.398655 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.398674 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.398973 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.398979 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.399027 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.399076 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20240501 02:07:37.400849 24461 raft_consensus.cc:1261] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Refusing update from remote peer b67fd7cc79f44c1d83a054d773f05d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:37.401150 24603 raft_consensus.cc:1261] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Refusing update from remote peer b67fd7cc79f44c1d83a054d773f05d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:37.401154 24831 consensus_queue.cc:1035] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:37.401404 24815 consensus_queue.cc:1035] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:37.402335 24817 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: TxnStatusTablet state changed. Reason: New leader b67fd7cc79f44c1d83a054d773f05d4e. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.402415 24817 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:37.402503 24816 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: TxnStatusTablet state changed. Reason: New leader b67fd7cc79f44c1d83a054d773f05d4e. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.402611 24816 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:37.403662 24831 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.403692 24815 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.403762 24831 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:07:37.403762 24815 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P b67fd7cc79f44c1d83a054d773f05d4e: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:07:37.404455 24837 mvcc.cc:205] Tried to move back new op lower bound from 7022711838311452672 to 7022711838136397824. Current Snapshot: MvccSnapshot[applied={T|T < 7022711838311452672}]
I20240501 02:07:37.404466 24817 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.404603 24817 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P 8ba839eb732d40d198674ef83297c008: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:37.405267 24816 tablet_replica.cc:439] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } } }
I20240501 02:07:37.405351 24816 tablet_replica.cc:442] T ea3dbab4d9204257b626be95c79bc6ca P f02930bf38144b73b5aa56d16428771e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:07:37.405545 24838 mvcc.cc:205] Tried to move back new op lower bound from 7022711838311452672 to 7022711838136397824. Current Snapshot: MvccSnapshot[applied={T|T < 7022711838311452672}]
I20240501 02:07:37.408282 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.405992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.22.12.66:43193)
0501 02:07:37.406033 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.406035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.406043 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:07:37.406078 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.406079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.406096 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.406184 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:37.406406 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.407165 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.407730 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.407897 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.407936 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.408039 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.408115 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.408184 (+ 69us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.408207 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:07:37.412704 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.410524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:39133 (local address 127.22.12.67:33757)
0501 02:07:37.410587 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.410590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.410699 (+ 109us) server_negotiation.cc:407] Connection header received
0501 02:07:37.410766 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.410767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.410776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.410819 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:37.411135 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.411551 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.412381 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.412496 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.412510 (+ 14us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.412532 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.412568 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.412635 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.412650 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:07:37.412761 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.410441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33757 (local address 127.22.12.66:39133)
0501 02:07:37.410628 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.410639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.410704 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.410833 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.410834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.410835 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:37.411078 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.411080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.411626 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.411627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.412312 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.412315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.412395 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.412411 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.412586 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.412587 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.412631 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.412672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":119,"threads_started":1}
I20240501 02:07:37.415118 24743 raft_consensus.cc:1261] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Refusing update from remote peer 8ba839eb732d40d198674ef83297c008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:37.415130 24461 raft_consensus.cc:1261] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Refusing update from remote peer 8ba839eb732d40d198674ef83297c008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:37.415359 24817 consensus_queue.cc:1035] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:37.415485 24819 consensus_queue.cc:1035] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:37.417670 24836 mvcc.cc:205] Tried to move back new op lower bound from 7022711838370426880 to 7022711836915286016. Current Snapshot: MvccSnapshot[applied={T|T < 7022711838370426880}]
I20240501 02:07:37.424038 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.421482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:34857 (local address 127.22.12.66:43193)
0501 02:07:37.421521 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:37.421523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:37.421694 (+ 171us) server_negotiation.cc:407] Connection header received
0501 02:07:37.421781 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:37.421782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:37.421792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:37.421847 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:37.422088 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.422833 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.423596 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.423769 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.423788 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.423816 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.423875 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:37.423937 (+ 62us) server_negotiation.cc:299] Negotiation successful
0501 02:07:37.423963 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:07:37.424050 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:37.421454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43193 (local address 127.22.12.67:34857)
0501 02:07:37.421627 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:37.421635 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:37.421730 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:37.421849 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:37.421849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:37.421850 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:37.422033 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.422034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.422938 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:37.422939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:37.423492 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:37.423494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:37.423567 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:37.423587 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:37.423879 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:37.423881 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:07:37.423917 (+ 36us) client_negotiation.cc:240] Negotiation successful
0501 02:07:37.423964 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20240501 02:07:37.430485 24305 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:37.430641 24305 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:37.431372 24305 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Sending DeleteTablet for 3 replicas of tablet e49079f49c3545459f3e6dbea6f3b7cd
I20240501 02:07:37.431666 24443 tablet_service.cc:1514] Processing DeleteTablet for tablet e49079f49c3545459f3e6dbea6f3b7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:37 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:37588
I20240501 02:07:37.431702 24709 tablet_service.cc:1514] Processing DeleteTablet for tablet e49079f49c3545459f3e6dbea6f3b7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:37 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57044
I20240501 02:07:37.431731 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet e49079f49c3545459f3e6dbea6f3b7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:37 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:32850
I20240501 02:07:37.432209 24856 tablet_replica.cc:330] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: stopping tablet replica
I20240501 02:07:37.432329 24305 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 02:07:37.432405 24856 raft_consensus.cc:2229] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 02:07:37.432479 24305 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:37.432775 24856 raft_consensus.cc:2258] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:37.433027 24857 tablet_replica.cc:330] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: stopping tablet replica
I20240501 02:07:37.433352 24856 ts_tablet_manager.cc:1875] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:37.433710 24857 raft_consensus.cc:2229] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:37.433816 24709 tablet_service.cc:1467] Processing CreateTablet for tablet b1163498b2d74e49b2829d60cc5cd3c0 (DEFAULT_TABLE table=test-table [id=f8d81727004b4e2bb35e5da9ee9b9871]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:37.433869 24443 tablet_service.cc:1467] Processing CreateTablet for tablet b1163498b2d74e49b2829d60cc5cd3c0 (DEFAULT_TABLE table=test-table [id=f8d81727004b4e2bb35e5da9ee9b9871]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:37.433940 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1163498b2d74e49b2829d60cc5cd3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:37.433948 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1163498b2d74e49b2829d60cc5cd3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:37.433863 24857 raft_consensus.cc:2258] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:37.434460 24857 ts_tablet_manager.cc:1875] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:37.435220 24809 tablet_bootstrap.cc:492] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Bootstrap starting.
I20240501 02:07:37.435345 24583 tablet_service.cc:1467] Processing CreateTablet for tablet b1163498b2d74e49b2829d60cc5cd3c0 (DEFAULT_TABLE table=test-table [id=f8d81727004b4e2bb35e5da9ee9b9871]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:37.435424 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1163498b2d74e49b2829d60cc5cd3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:37.435595 24809 tablet_bootstrap.cc:654] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:37.436192 24809 tablet_bootstrap.cc:492] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: No bootstrap required, opened a new log
I20240501 02:07:37.436245 24809 ts_tablet_manager.cc:1367] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:37.436338 24810 tablet_bootstrap.cc:492] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Bootstrap starting.
I20240501 02:07:37.436395 24809 raft_consensus.cc:348] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.436466 24809 raft_consensus.cc:374] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:37.436486 24809 raft_consensus.cc:729] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67fd7cc79f44c1d83a054d773f05d4e, State: Initialized, Role: FOLLOWER
I20240501 02:07:37.436532 24809 consensus_queue.cc:260] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.436622 24809 ts_tablet_manager.cc:1398] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:37.436720 24810 tablet_bootstrap.cc:654] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:37.437016 24811 tablet_bootstrap.cc:492] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Bootstrap starting.
I20240501 02:07:37.437366 24811 tablet_bootstrap.cc:654] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:37.438000 24811 tablet_bootstrap.cc:492] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: No bootstrap required, opened a new log
I20240501 02:07:37.438043 24811 ts_tablet_manager.cc:1367] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240501 02:07:37.438179 24811 raft_consensus.cc:348] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.438236 24811 raft_consensus.cc:374] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:37.438277 24811 raft_consensus.cc:729] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Initialized, Role: FOLLOWER
I20240501 02:07:37.438318 24811 consensus_queue.cc:260] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.438371 24857 ts_tablet_manager.cc:1888] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:37.438402 24811 ts_tablet_manager.cc:1398] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:37.438438 24857 log.cc:1196] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/e49079f49c3545459f3e6dbea6f3b7cd
I20240501 02:07:37.438724 24857 ts_tablet_manager.cc:1909] T e49079f49c3545459f3e6dbea6f3b7cd P 8ba839eb732d40d198674ef83297c008: Deleting consensus metadata
I20240501 02:07:37.438894 24858 tablet_replica.cc:330] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: stopping tablet replica
I20240501 02:07:37.438982 24858 raft_consensus.cc:2229] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:37.439040 24858 raft_consensus.cc:2258] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:37.439327 24293 catalog_manager.cc:4830] TS 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193): tablet e49079f49c3545459f3e6dbea6f3b7cd (table test-table [id=4e3a9bf8f46040b58327378dda901f35]) successfully deleted
I20240501 02:07:37.439400 24858 ts_tablet_manager.cc:1875] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:37.442153 24858 ts_tablet_manager.cc:1888] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:37.442224 24858 log.cc:1196] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/e49079f49c3545459f3e6dbea6f3b7cd
I20240501 02:07:37.442461 24858 ts_tablet_manager.cc:1909] T e49079f49c3545459f3e6dbea6f3b7cd P b67fd7cc79f44c1d83a054d773f05d4e: Deleting consensus metadata
I20240501 02:07:37.442945 24291 catalog_manager.cc:4830] TS b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757): tablet e49079f49c3545459f3e6dbea6f3b7cd (table test-table [id=4e3a9bf8f46040b58327378dda901f35]) successfully deleted
I20240501 02:07:37.446765 24856 ts_tablet_manager.cc:1888] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:37.446857 24856 log.cc:1196] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/e49079f49c3545459f3e6dbea6f3b7cd
I20240501 02:07:37.447119 24856 ts_tablet_manager.cc:1909] T e49079f49c3545459f3e6dbea6f3b7cd P f02930bf38144b73b5aa56d16428771e: Deleting consensus metadata
I20240501 02:07:37.447613 24810 tablet_bootstrap.cc:492] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: No bootstrap required, opened a new log
I20240501 02:07:37.447669 24810 ts_tablet_manager.cc:1367] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 02:07:37.447830 24290 catalog_manager.cc:4830] TS f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073): tablet e49079f49c3545459f3e6dbea6f3b7cd (table test-table [id=4e3a9bf8f46040b58327378dda901f35]) successfully deleted
I20240501 02:07:37.447824 24810 raft_consensus.cc:348] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.447893 24810 raft_consensus.cc:374] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:37.447914 24810 raft_consensus.cc:729] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02930bf38144b73b5aa56d16428771e, State: Initialized, Role: FOLLOWER
I20240501 02:07:37.447966 24810 consensus_queue.cc:260] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.448062 24810 ts_tablet_manager.cc:1398] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:37.608197 24854 raft_consensus.cc:482] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:37.608302 24854 raft_consensus.cc:504] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.608446 24854 leader_election.cc:288] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:37.608682 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1163498b2d74e49b2829d60cc5cd3c0" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e" is_pre_election: true
I20240501 02:07:37.608685 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1163498b2d74e49b2829d60cc5cd3c0" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" is_pre_election: true
I20240501 02:07:37.608767 24461 raft_consensus.cc:2454] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 0.
I20240501 02:07:37.608780 24743 raft_consensus.cc:2454] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 0.
I20240501 02:07:37.608913 24534 leader_election.cc:302] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:37.609025 24854 raft_consensus.cc:2781] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:37.609058 24854 raft_consensus.cc:482] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:37.609078 24854 raft_consensus.cc:3037] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.609767 24854 raft_consensus.cc:504] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.609905 24854 leader_election.cc:288] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 election: Requested vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:37.610077 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1163498b2d74e49b2829d60cc5cd3c0" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
I20240501 02:07:37.610077 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1163498b2d74e49b2829d60cc5cd3c0" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e"
I20240501 02:07:37.610157 24743 raft_consensus.cc:3037] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.610157 24461 raft_consensus.cc:3037] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:37.610965 24461 raft_consensus.cc:2454] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 1.
I20240501 02:07:37.610965 24743 raft_consensus.cc:2454] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 1.
I20240501 02:07:37.611160 24534 leader_election.cc:302] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:37.611297 24854 raft_consensus.cc:2781] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:37.611352 24854 raft_consensus.cc:686] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Running, Role: LEADER
I20240501 02:07:37.611411 24854 consensus_queue.cc:237] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:37.612006 24305 catalog_manager.cc:5484] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba839eb732d40d198674ef83297c008 (127.22.12.66). New cstate: current_term: 1 leader_uuid: "8ba839eb732d40d198674ef83297c008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } health_report { overall_health: UNKNOWN } } }
I20240501 02:07:37.621224 24743 raft_consensus.cc:1261] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Refusing update from remote peer 8ba839eb732d40d198674ef83297c008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:37.621270 24461 raft_consensus.cc:1261] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Refusing update from remote peer 8ba839eb732d40d198674ef83297c008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:37.621455 24817 consensus_queue.cc:1035] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:37.621562 24854 consensus_queue.cc:1035] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 02:07:37.627025 24837 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240501 02:07:39.156679 24306 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:39.156777 24306 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40998:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:39.157549 24306 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Sending DeleteTablet for 3 replicas of tablet b1163498b2d74e49b2829d60cc5cd3c0
I20240501 02:07:39.157800 24443 tablet_service.cc:1514] Processing DeleteTablet for tablet b1163498b2d74e49b2829d60cc5cd3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:39 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:37588
I20240501 02:07:39.157824 24709 tablet_service.cc:1514] Processing DeleteTablet for tablet b1163498b2d74e49b2829d60cc5cd3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:39 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57044
I20240501 02:07:39.158051 24888 tablet_replica.cc:330] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: stopping tablet replica
I20240501 02:07:39.158140 24888 raft_consensus.cc:2229] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:39.158196 24888 raft_consensus.cc:2258] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:39.159298 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet b1163498b2d74e49b2829d60cc5cd3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:39 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:32850
I20240501 02:07:39.159971 24888 ts_tablet_manager.cc:1875] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:39.161515 24888 ts_tablet_manager.cc:1888] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 02:07:39.161566 24888 log.cc:1196] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/b1163498b2d74e49b2829d60cc5cd3c0
I20240501 02:07:39.161794 24888 ts_tablet_manager.cc:1909] T b1163498b2d74e49b2829d60cc5cd3c0 P f02930bf38144b73b5aa56d16428771e: Deleting consensus metadata
I20240501 02:07:39.162089 24290 catalog_manager.cc:4830] TS f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073): tablet b1163498b2d74e49b2829d60cc5cd3c0 (table test-table [id=f8d81727004b4e2bb35e5da9ee9b9871]) successfully deleted
I20240501 02:07:39.162623 24890 tablet_replica.cc:330] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: stopping tablet replica
I20240501 02:07:39.162688 24890 raft_consensus.cc:2229] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:39.162761 24890 raft_consensus.cc:2258] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:39.162869 24887 tablet_replica.cc:330] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: stopping tablet replica
I20240501 02:07:39.162937 24887 raft_consensus.cc:2229] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:39.162982 24887 raft_consensus.cc:2258] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:39.163056 24890 ts_tablet_manager.cc:1875] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:39.163199 24887 ts_tablet_manager.cc:1875] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:39.164728 24890 ts_tablet_manager.cc:1888] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 02:07:39.164786 24890 log.cc:1196] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/b1163498b2d74e49b2829d60cc5cd3c0
I20240501 02:07:39.164767 24887 ts_tablet_manager.cc:1888] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 02:07:39.164819 24887 log.cc:1196] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/b1163498b2d74e49b2829d60cc5cd3c0
I20240501 02:07:39.165021 24890 ts_tablet_manager.cc:1909] T b1163498b2d74e49b2829d60cc5cd3c0 P 8ba839eb732d40d198674ef83297c008: Deleting consensus metadata
I20240501 02:07:39.165112 24887 ts_tablet_manager.cc:1909] T b1163498b2d74e49b2829d60cc5cd3c0 P b67fd7cc79f44c1d83a054d773f05d4e: Deleting consensus metadata
I20240501 02:07:39.165349 24293 catalog_manager.cc:4830] TS 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193): tablet b1163498b2d74e49b2829d60cc5cd3c0 (table test-table [id=f8d81727004b4e2bb35e5da9ee9b9871]) successfully deleted
I20240501 02:07:39.165385 24291 catalog_manager.cc:4830] TS b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757): tablet b1163498b2d74e49b2829d60cc5cd3c0 (table test-table [id=f8d81727004b4e2bb35e5da9ee9b9871]) successfully deleted
May 01 02:07:39 dist-test-slave-9tkh krb5kdc[24261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:07:39 dist-test-slave-9tkh krb5kdc[24261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:39.183347 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.175978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.22.12.126:36261)
0501 02:07:39.176166 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.176167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.176183 (+ 16us) server_negotiation.cc:407] Connection header received
0501 02:07:39.176289 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.176290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.176302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.176353 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:39.177098 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.177657 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.178204 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.178372 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.180585 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:39.180591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:39.180592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:39.180599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:39.182080 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:39.182444 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:39.182445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:39.182446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:39.182473 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:39.182745 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:39.182747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:39.182747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:39.182874 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:39.182985 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.183165 (+ 180us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.183267 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":113,"threads_started":1}
I20240501 02:07:39.185115 24306 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:39.185276 24306 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:39.186875 24443 tablet_service.cc:1467] Processing CreateTablet for tablet 6528abeec4364ff180ee1306114003ea (DEFAULT_TABLE table=test-table [id=889f449ee6384de39bd326e26872b4bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:39.187047 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6528abeec4364ff180ee1306114003ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:39.188055 24898 tablet_bootstrap.cc:492] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Bootstrap starting.
I20240501 02:07:39.188534 24898 tablet_bootstrap.cc:654] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:39.189386 24898 tablet_bootstrap.cc:492] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: No bootstrap required, opened a new log
I20240501 02:07:39.189431 24898 ts_tablet_manager.cc:1367] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:39.189572 24898 raft_consensus.cc:348] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.189633 24898 raft_consensus.cc:374] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:39.189646 24898 raft_consensus.cc:729] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02930bf38144b73b5aa56d16428771e, State: Initialized, Role: FOLLOWER
I20240501 02:07:39.189702 24898 consensus_queue.cc:260] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.189865 24898 ts_tablet_manager.cc:1398] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:39.190724 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 6528abeec4364ff180ee1306114003ea (DEFAULT_TABLE table=test-table [id=889f449ee6384de39bd326e26872b4bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:39.188099 24709 tablet_service.cc:1467] Processing CreateTablet for tablet 6528abeec4364ff180ee1306114003ea (DEFAULT_TABLE table=test-table [id=889f449ee6384de39bd326e26872b4bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:39.190974 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6528abeec4364ff180ee1306114003ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:39.193194 24901 tablet_bootstrap.cc:492] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Bootstrap starting.
I20240501 02:07:39.193282 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6528abeec4364ff180ee1306114003ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:39.193744 24901 tablet_bootstrap.cc:654] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:39.194628 24903 tablet_bootstrap.cc:492] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Bootstrap starting.
I20240501 02:07:39.195045 24903 tablet_bootstrap.cc:654] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:39.195070 24901 tablet_bootstrap.cc:492] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: No bootstrap required, opened a new log
I20240501 02:07:39.195117 24901 ts_tablet_manager.cc:1367] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:39.195290 24901 raft_consensus.cc:348] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.195428 24901 raft_consensus.cc:374] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:39.195469 24901 raft_consensus.cc:729] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67fd7cc79f44c1d83a054d773f05d4e, State: Initialized, Role: FOLLOWER
I20240501 02:07:39.195540 24901 consensus_queue.cc:260] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.195662 24903 tablet_bootstrap.cc:492] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: No bootstrap required, opened a new log
I20240501 02:07:39.195715 24903 ts_tablet_manager.cc:1367] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:39.195791 24901 ts_tablet_manager.cc:1398] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 02:07:39.195842 24903 raft_consensus.cc:348] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.195914 24903 raft_consensus.cc:374] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:39.195940 24903 raft_consensus.cc:729] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Initialized, Role: FOLLOWER
I20240501 02:07:39.195986 24903 consensus_queue.cc:260] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.196087 24903 ts_tablet_manager.cc:1398] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:39.196391 24900 raft_consensus.cc:482] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:39.196506 24900 raft_consensus.cc:504] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.197284 24900 leader_election.cc:288] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:39.200181 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.196992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33757 (local address 127.22.12.65:53225)
0501 02:07:39.197150 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:39.197160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:39.197214 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:39.197567 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:39.197567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:39.197569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:39.197826 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.197828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.198895 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.198896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:39.199912 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.199915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.199996 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.200002 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:07:39.200036 (+ 34us) client_negotiation.cc:240] Negotiation successful
0501 02:07:39.200084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20240501 02:07:39.200191 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.196992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43193 (local address 127.22.12.65:59289)
0501 02:07:39.197401 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:39.197410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:39.197490 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:39.197693 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:39.197694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:39.197695 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:39.197914 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.197915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.198875 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.198877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:39.199912 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.199915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.199998 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.200002 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:39.200032 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 02:07:39.200076 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":11,"thread_start_us":84,"threads_started":1}
I20240501 02:07:39.200701 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.197099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:59289 (local address 127.22.12.66:43193)
0501 02:07:39.197582 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.197584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.197596 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:07:39.197615 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.197616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.197626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.197692 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:39.197987 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.198766 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.200022 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.200578 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.200585 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.200606 (+ 21us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.200634 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":178,"threads_started":1}
I20240501 02:07:39.200713 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.197188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:53225 (local address 127.22.12.67:33757)
0501 02:07:39.197462 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.197464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.197474 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:39.197496 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.197497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.197507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.197556 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:39.197905 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.198779 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.200066 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.200564 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.200575 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.200600 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.200632 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
I20240501 02:07:39.200896 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6528abeec4364ff180ee1306114003ea" candidate_uuid: "f02930bf38144b73b5aa56d16428771e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba839eb732d40d198674ef83297c008" is_pre_election: true
I20240501 02:07:39.200968 24603 raft_consensus.cc:2454] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f02930bf38144b73b5aa56d16428771e in term 0.
I20240501 02:07:39.201131 24397 leader_election.cc:302] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:39.201252 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6528abeec4364ff180ee1306114003ea" candidate_uuid: "f02930bf38144b73b5aa56d16428771e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" is_pre_election: true
I20240501 02:07:39.201296 24900 raft_consensus.cc:2781] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:39.201349 24743 raft_consensus.cc:2454] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f02930bf38144b73b5aa56d16428771e in term 0.
I20240501 02:07:39.201361 24900 raft_consensus.cc:482] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:39.201382 24900 raft_consensus.cc:3037] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:39.202060 24900 raft_consensus.cc:504] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.202198 24900 leader_election.cc:288] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [CANDIDATE]: Term 1 election: Requested vote from peers 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:39.202387 24603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6528abeec4364ff180ee1306114003ea" candidate_uuid: "f02930bf38144b73b5aa56d16428771e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba839eb732d40d198674ef83297c008"
I20240501 02:07:39.202421 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6528abeec4364ff180ee1306114003ea" candidate_uuid: "f02930bf38144b73b5aa56d16428771e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
I20240501 02:07:39.202469 24603 raft_consensus.cc:3037] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:39.202477 24743 raft_consensus.cc:3037] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:39.203261 24603 raft_consensus.cc:2454] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f02930bf38144b73b5aa56d16428771e in term 1.
I20240501 02:07:39.203398 24743 raft_consensus.cc:2454] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f02930bf38144b73b5aa56d16428771e in term 1.
I20240501 02:07:39.203480 24397 leader_election.cc:302] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:39.203588 24900 raft_consensus.cc:2781] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:39.203735 24900 raft_consensus.cc:686] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 1 LEADER]: Becoming Leader. State: Replica: f02930bf38144b73b5aa56d16428771e, State: Running, Role: LEADER
I20240501 02:07:39.203809 24900 consensus_queue.cc:237] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.204613 24306 catalog_manager.cc:5484] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e reported cstate change: term changed from 0 to 1, leader changed from <none> to f02930bf38144b73b5aa56d16428771e (127.22.12.65). New cstate: current_term: 1 leader_uuid: "f02930bf38144b73b5aa56d16428771e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } health_report { overall_health: UNKNOWN } } }
I20240501 02:07:39.216850 24912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.214710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.22.12.65:38073)
0501 02:07:39.214829 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.214831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.214843 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:07:39.214878 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.214879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.214896 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.214972 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:39.215247 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.215761 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.216319 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.216447 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.216488 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:39.216594 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:39.216654 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.216747 (+ 93us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.216767 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20240501 02:07:39.220963 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.219157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33757 (local address 127.22.12.65:37585)
0501 02:07:39.219314 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:39.219323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:39.219376 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:39.219484 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:39.219485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:39.219486 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:39.219659 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.219660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.220189 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.220190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:39.220590 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.220591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.220651 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.220666 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:39.220829 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:39.220830 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:07:39.220862 (+ 32us) client_negotiation.cc:240] Negotiation successful
0501 02:07:39.220893 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20240501 02:07:39.220983 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.219214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:37585 (local address 127.22.12.67:33757)
0501 02:07:39.219267 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.219269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.219374 (+ 105us) server_negotiation.cc:407] Connection header received
0501 02:07:39.219414 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.219414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.219423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.219480 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:39.219706 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.220139 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.220648 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.220749 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.220763 (+ 14us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:39.220784 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:39.220817 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.220903 (+ 86us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.220928 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 02:07:39.223080 24743 raft_consensus.cc:1261] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Refusing update from remote peer f02930bf38144b73b5aa56d16428771e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:39.223119 24603 raft_consensus.cc:1261] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Refusing update from remote peer f02930bf38144b73b5aa56d16428771e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:39.223366 24909 consensus_queue.cc:1035] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:39.223495 24900 consensus_queue.cc:1035] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:39.230417 24912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.228433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:47345 (local address 127.22.12.65:38073)
0501 02:07:39.228476 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.228478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.228557 (+ 79us) server_negotiation.cc:407] Connection header received
0501 02:07:39.228625 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.228626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.228638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.228690 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:39.228986 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.229462 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.229997 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.230143 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.230165 (+ 22us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:39.230194 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:39.230237 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.230316 (+ 79us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.230339 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:07:39.230422 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.228365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38073 (local address 127.22.12.67:47345)
0501 02:07:39.228493 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:39.228503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:39.228566 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:39.228751 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:39.228752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:39.228754 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:07:39.228924 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.228926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.229537 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.229538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:39.229928 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:39.229929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.229987 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.229993 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:07:39.230258 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:07:39.230260 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:07:39.230296 (+ 36us) client_negotiation.cc:240] Negotiation successful
0501 02:07:39.230345 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20240501 02:07:39.237439 24305 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41014:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:39.237548 24305 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41014:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:39.238198 24305 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Sending DeleteTablet for 3 replicas of tablet 6528abeec4364ff180ee1306114003ea
I20240501 02:07:39.238430 24443 tablet_service.cc:1514] Processing DeleteTablet for tablet 6528abeec4364ff180ee1306114003ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:39 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:37588
I20240501 02:07:39.238571 24888 tablet_replica.cc:330] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: stopping tablet replica
I20240501 02:07:39.238564 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet 6528abeec4364ff180ee1306114003ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:39 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:32850
I20240501 02:07:39.238616 24888 raft_consensus.cc:2229] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:39.238667 24890 tablet_replica.cc:330] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: stopping tablet replica
I20240501 02:07:39.238704 24888 raft_consensus.cc:2258] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:39.238709 24890 raft_consensus.cc:2229] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:39.238744 24890 raft_consensus.cc:2258] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:39.238747 24305 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:39.238881 24305 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:39.238886 24709 tablet_service.cc:1514] Processing DeleteTablet for tablet 6528abeec4364ff180ee1306114003ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:39 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57044
I20240501 02:07:39.238977 24887 tablet_replica.cc:330] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: stopping tablet replica
I20240501 02:07:39.239034 24887 raft_consensus.cc:2229] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:39.239077 24887 raft_consensus.cc:2258] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:39.239171 24890 ts_tablet_manager.cc:1875] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:39.239260 24888 ts_tablet_manager.cc:1875] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:39.240293 24583 tablet_service.cc:1467] Processing CreateTablet for tablet f6adc99639fa46048f15ecff2986a985 (DEFAULT_TABLE table=test-table [id=575c589b2d3c4e51bbc0bda8b9f12983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:39.240392 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6adc99639fa46048f15ecff2986a985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:39.240744 24443 tablet_service.cc:1467] Processing CreateTablet for tablet f6adc99639fa46048f15ecff2986a985 (DEFAULT_TABLE table=test-table [id=575c589b2d3c4e51bbc0bda8b9f12983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:39.240849 24709 tablet_service.cc:1467] Processing CreateTablet for tablet f6adc99639fa46048f15ecff2986a985 (DEFAULT_TABLE table=test-table [id=575c589b2d3c4e51bbc0bda8b9f12983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:39.240945 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6adc99639fa46048f15ecff2986a985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:39.241007 24887 ts_tablet_manager.cc:1875] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:39.241021 24890 ts_tablet_manager.cc:1888] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:39.241078 24890 log.cc:1196] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/6528abeec4364ff180ee1306114003ea
I20240501 02:07:39.241277 24890 ts_tablet_manager.cc:1909] T 6528abeec4364ff180ee1306114003ea P 8ba839eb732d40d198674ef83297c008: Deleting consensus metadata
I20240501 02:07:39.241585 24903 tablet_bootstrap.cc:492] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Bootstrap starting.
I20240501 02:07:39.241889 24903 tablet_bootstrap.cc:654] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:39.242213 24901 tablet_bootstrap.cc:492] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Bootstrap starting.
I20240501 02:07:39.242312 24293 catalog_manager.cc:4830] TS 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193): tablet 6528abeec4364ff180ee1306114003ea (table test-table [id=889f449ee6384de39bd326e26872b4bd]) successfully deleted
I20240501 02:07:39.242487 24888 ts_tablet_manager.cc:1888] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:39.242492 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6adc99639fa46048f15ecff2986a985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:39.242547 24901 tablet_bootstrap.cc:654] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:39.242561 24888 log.cc:1196] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/6528abeec4364ff180ee1306114003ea
I20240501 02:07:39.242766 24888 ts_tablet_manager.cc:1909] T 6528abeec4364ff180ee1306114003ea P f02930bf38144b73b5aa56d16428771e: Deleting consensus metadata
I20240501 02:07:39.242794 24887 ts_tablet_manager.cc:1888] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:07:39.242838 24887 log.cc:1196] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/6528abeec4364ff180ee1306114003ea
I20240501 02:07:39.243055 24887 ts_tablet_manager.cc:1909] T 6528abeec4364ff180ee1306114003ea P b67fd7cc79f44c1d83a054d773f05d4e: Deleting consensus metadata
I20240501 02:07:39.243089 24290 catalog_manager.cc:4830] TS f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073): tablet 6528abeec4364ff180ee1306114003ea (table test-table [id=889f449ee6384de39bd326e26872b4bd]) successfully deleted
I20240501 02:07:39.243232 24903 tablet_bootstrap.cc:492] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: No bootstrap required, opened a new log
I20240501 02:07:39.243274 24903 ts_tablet_manager.cc:1367] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:39.243367 24901 tablet_bootstrap.cc:492] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: No bootstrap required, opened a new log
I20240501 02:07:39.243415 24901 ts_tablet_manager.cc:1367] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:39.243403 24903 raft_consensus.cc:348] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.243468 24903 raft_consensus.cc:374] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:39.243491 24903 raft_consensus.cc:729] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Initialized, Role: FOLLOWER
I20240501 02:07:39.243530 24903 consensus_queue.cc:260] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.243517 24901 raft_consensus.cc:348] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.243577 24901 raft_consensus.cc:374] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:39.243602 24901 raft_consensus.cc:729] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67fd7cc79f44c1d83a054d773f05d4e, State: Initialized, Role: FOLLOWER
I20240501 02:07:39.243623 24903 ts_tablet_manager.cc:1398] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:39.243639 24901 consensus_queue.cc:260] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.243733 24901 ts_tablet_manager.cc:1398] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:39.243814 24898 tablet_bootstrap.cc:492] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e: Bootstrap starting.
I20240501 02:07:39.244179 24898 tablet_bootstrap.cc:654] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:39.244571 24291 catalog_manager.cc:4830] TS b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757): tablet 6528abeec4364ff180ee1306114003ea (table test-table [id=889f449ee6384de39bd326e26872b4bd]) successfully deleted
I20240501 02:07:39.244886 24898 tablet_bootstrap.cc:492] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e: No bootstrap required, opened a new log
I20240501 02:07:39.244940 24898 ts_tablet_manager.cc:1367] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:39.245100 24898 raft_consensus.cc:348] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.245175 24898 raft_consensus.cc:374] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:39.245195 24898 raft_consensus.cc:729] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02930bf38144b73b5aa56d16428771e, State: Initialized, Role: FOLLOWER
I20240501 02:07:39.245244 24898 consensus_queue.cc:260] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.245333 24898 ts_tablet_manager.cc:1398] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:39.280120 24878 raft_consensus.cc:482] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:39.280238 24878 raft_consensus.cc:504] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.280417 24878 leader_election.cc:288] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:39.280632 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6adc99639fa46048f15ecff2986a985" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e" is_pre_election: true
I20240501 02:07:39.280725 24461 raft_consensus.cc:2454] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 0.
I20240501 02:07:39.280828 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6adc99639fa46048f15ecff2986a985" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" is_pre_election: true
I20240501 02:07:39.280911 24743 raft_consensus.cc:2454] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 0.
I20240501 02:07:39.280887 24534 leader_election.cc:302] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, f02930bf38144b73b5aa56d16428771e; no voters:
I20240501 02:07:39.281014 24878 raft_consensus.cc:2781] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:39.281073 24878 raft_consensus.cc:482] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:39.281100 24878 raft_consensus.cc:3037] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:39.281740 24878 raft_consensus.cc:504] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.281875 24878 leader_election.cc:288] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 election: Requested vote from peers f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073), b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757)
I20240501 02:07:39.282034 24743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6adc99639fa46048f15ecff2986a985" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67fd7cc79f44c1d83a054d773f05d4e"
I20240501 02:07:39.282120 24743 raft_consensus.cc:3037] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:39.282100 24461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6adc99639fa46048f15ecff2986a985" candidate_uuid: "8ba839eb732d40d198674ef83297c008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02930bf38144b73b5aa56d16428771e"
I20240501 02:07:39.282183 24461 raft_consensus.cc:3037] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:39.282753 24743 raft_consensus.cc:2454] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 1.
I20240501 02:07:39.282950 24535 leader_election.cc:302] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba839eb732d40d198674ef83297c008, b67fd7cc79f44c1d83a054d773f05d4e; no voters:
I20240501 02:07:39.283004 24461 raft_consensus.cc:2454] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba839eb732d40d198674ef83297c008 in term 1.
I20240501 02:07:39.283077 24878 raft_consensus.cc:2781] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:39.283133 24878 raft_consensus.cc:686] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba839eb732d40d198674ef83297c008, State: Running, Role: LEADER
I20240501 02:07:39.283186 24878 consensus_queue.cc:237] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } }
I20240501 02:07:39.283753 24306 catalog_manager.cc:5484] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba839eb732d40d198674ef83297c008 (127.22.12.66). New cstate: current_term: 1 leader_uuid: "8ba839eb732d40d198674ef83297c008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba839eb732d40d198674ef83297c008" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 } health_report { overall_health: UNKNOWN } } }
I20240501 02:07:39.292366 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:39.290019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.22.12.66:43193)
0501 02:07:39.290068 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:39.290070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:39.290092 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:07:39.290163 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:39.290164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:39.290172 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:39.290217 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:39.290508 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.291115 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:39.291957 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:39.292107 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:39.292125 (+ 18us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:39.292149 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:39.292199 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:39.292264 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 02:07:39.292283 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:07:39.294566 24743 raft_consensus.cc:1261] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Refusing update from remote peer 8ba839eb732d40d198674ef83297c008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:39.294523 24461 raft_consensus.cc:1261] T f6adc99639fa46048f15ecff2986a985 P f02930bf38144b73b5aa56d16428771e [term 1 FOLLOWER]: Refusing update from remote peer 8ba839eb732d40d198674ef83297c008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:39.294803 24881 consensus_queue.cc:1035] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02930bf38144b73b5aa56d16428771e" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:39.294926 24878 consensus_queue.cc:1035] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67fd7cc79f44c1d83a054d773f05d4e" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 02:07:39.300448 24869 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240501 02:07:40.818341 24306 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41014:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:40.818441 24306 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41014:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:40.819430 24306 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6925a926955d4c7e8194a666cff54325: Sending DeleteTablet for 3 replicas of tablet f6adc99639fa46048f15ecff2986a985
I20240501 02:07:40.819682 24443 tablet_service.cc:1514] Processing DeleteTablet for tablet f6adc99639fa46048f15ecff2986a985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:40 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:37588
I20240501 02:07:40.819690 24723 tablet_service.cc:1514] Processing DeleteTablet for tablet f6adc99639fa46048f15ecff2986a985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:40 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57044
I20240501 02:07:40.819727 24583 tablet_service.cc:1514] Processing DeleteTablet for tablet f6adc99639fa46048f15ecff2986a985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:40 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:32850
I20240501 02:07:40.819895 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 24378
I20240501 02:07:40.821048 24954 tablet_replica.cc:330] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: stopping tablet replica
I20240501 02:07:40.821116 24954 raft_consensus.cc:2229] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:40.821182 24954 raft_consensus.cc:2258] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:40.821403 24954 ts_tablet_manager.cc:1875] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:40.823004 24954 ts_tablet_manager.cc:1888] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 02:07:40.823060 24954 log.cc:1196] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/f6adc99639fa46048f15ecff2986a985
I20240501 02:07:40.823257 24954 ts_tablet_manager.cc:1909] T f6adc99639fa46048f15ecff2986a985 P 8ba839eb732d40d198674ef83297c008: Deleting consensus metadata
I20240501 02:07:40.823643 24293 catalog_manager.cc:4830] TS 8ba839eb732d40d198674ef83297c008 (127.22.12.66:43193): tablet f6adc99639fa46048f15ecff2986a985 (table test-table [id=575c589b2d3c4e51bbc0bda8b9f12983]) successfully deleted
I20240501 02:07:40.826272 24953 tablet_replica.cc:330] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: stopping tablet replica
I20240501 02:07:40.826336 24953 raft_consensus.cc:2229] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:40.826385 24953 raft_consensus.cc:2258] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:40.826668 24953 ts_tablet_manager.cc:1875] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 02:07:40.826721 24290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:38073 (error 108)
I20240501 02:07:40.826998 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 24518
I20240501 02:07:40.827690 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:40.827188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38073 (local address 127.0.0.1:37616)
0501 02:07:40.827611 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:40.827652 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":66,"threads_started":1}
W20240501 02:07:40.827773 24290 catalog_manager.cc:4573] TS f02930bf38144b73b5aa56d16428771e (127.22.12.65:38073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6adc99639fa46048f15ecff2986a985: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38073: connect: Connection refused (error 111)
I20240501 02:07:40.831830 24953 ts_tablet_manager.cc:1888] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 02:07:40.831889 24953 log.cc:1196] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/f6adc99639fa46048f15ecff2986a985
I20240501 02:07:40.832115 24953 ts_tablet_manager.cc:1909] T f6adc99639fa46048f15ecff2986a985 P b67fd7cc79f44c1d83a054d773f05d4e: Deleting consensus metadata
I20240501 02:07:40.832444 24291 catalog_manager.cc:4830] TS b67fd7cc79f44c1d83a054d773f05d4e (127.22.12.67:33757): tablet f6adc99639fa46048f15ecff2986a985 (table test-table [id=575c589b2d3c4e51bbc0bda8b9f12983]) successfully deleted
I20240501 02:07:40.832516 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 24658
I20240501 02:07:40.837451 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 24274
2024-05-01T02:07:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5590 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:40 dist-test-slave-9tkh krb5kdc[24959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:40 dist-test-slave-9tkh krb5kdc[24959](info): set up 2 sockets
May 01 02:07:40 dist-test-slave-9tkh krb5kdc[24959](info): commencing operation
krb5kdc: starting...
W20240501 02:07:41.738222 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:41 dist-test-slave-9tkh krb5kdc[24959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:41.848937 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41867
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:41.926410 24972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:41.926658 24972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:41.926692 24972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:41.928026 24972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:41.928077 24972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:41.928095 24972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:41.928112 24972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:41.928125 24972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:41.929733 24972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:41.929966 24972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:41.930153 24972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:41.934362 24978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:41.934700 24980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:41.935195 24977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:41.935600 24972 server_base.cc:1034] running on GCE node
I20240501 02:07:41.935832 24972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:41.936069 24972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:41.937222 24972 hybrid_clock.cc:648] HybridClock initialized: now 1714529261937199 us; error 36 us; skew 500 ppm
May 01 02:07:41 dist-test-slave-9tkh krb5kdc[24959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529261, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:41.944675 24972 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:41.945109 24972 webserver.cc:460] Webserver started at http://127.22.12.126:33709/ using document root <none> and password file <none>
I20240501 02:07:41.945291 24972 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:41.945331 24972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:41.945432 24972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:41.946394 24972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "3dc0de62dcc141a6a16cde458fe09bdc"
format_stamp: "Formatted at 2024-05-01 02:07:41 on dist-test-slave-9tkh"
server_key: "faaf1dafec6b34951ae8f1eb52422b9a"
server_key_iv: "49499b33e6c656bed67c58b86553c4bc"
server_key_version: "encryptionkey@0"
I20240501 02:07:41.946708 24972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "3dc0de62dcc141a6a16cde458fe09bdc"
format_stamp: "Formatted at 2024-05-01 02:07:41 on dist-test-slave-9tkh"
server_key: "faaf1dafec6b34951ae8f1eb52422b9a"
server_key_iv: "49499b33e6c656bed67c58b86553c4bc"
server_key_version: "encryptionkey@0"
I20240501 02:07:41.947824 24972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:41.948442 24987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:41.948604 24972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:41.948681 24972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "3dc0de62dcc141a6a16cde458fe09bdc"
format_stamp: "Formatted at 2024-05-01 02:07:41 on dist-test-slave-9tkh"
server_key: "faaf1dafec6b34951ae8f1eb52422b9a"
server_key_iv: "49499b33e6c656bed67c58b86553c4bc"
server_key_version: "encryptionkey@0"
I20240501 02:07:41.948737 24972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:41.965046 24972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:41.967180 24972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:41.967295 24972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:41.970973 24972 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41867
I20240501 02:07:41.971011 25039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41867 every 8 connection(s)
I20240501 02:07:41.971309 24972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:41.972041 25040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:41.973083 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 24972
I20240501 02:07:41.973166 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:41.973312 22577 external_mini_cluster.cc:1428] Setting key d0853785c6411ebf30c2dbc1786801b0
I20240501 02:07:41.974318 25040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc: Bootstrap starting.
I20240501 02:07:41.974875 25040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:41.975106 25040 log.cc:824] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:41.975714 25040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc: No bootstrap required, opened a new log
I20240501 02:07:41.977363 25040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41867 } }
I20240501 02:07:41.977496 25040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:41.977520 25040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc0de62dcc141a6a16cde458fe09bdc, State: Initialized, Role: FOLLOWER
I20240501 02:07:41.977617 25040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41867 } }
I20240501 02:07:41.977684 25040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:41.977715 25040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:41.977759 25040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:41.978302 25040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41867 } }
I20240501 02:07:41.978415 25040 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc0de62dcc141a6a16cde458fe09bdc; no voters:
I20240501 02:07:41.978600 25040 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:41.978642 25045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:41.978803 25045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [term 1 LEADER]: Becoming Leader. State: Replica: 3dc0de62dcc141a6a16cde458fe09bdc, State: Running, Role: LEADER
May 01 02:07:41 dist-test-slave-9tkh krb5kdc[24959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:41.978852 25040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:41.978961 25045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41867 } }
I20240501 02:07:41.979364 25046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc0de62dcc141a6a16cde458fe09bdc. Latest consensus state: current_term: 1 leader_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41867 } } }
I20240501 02:07:41.979524 25046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:41.979787 25054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:41.980309 25054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:41.980500 25047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc0de62dcc141a6a16cde458fe09bdc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41867 } } }
I20240501 02:07:41.980669 25047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:41.981725 25054 catalog_manager.cc:1347] Generated new cluster ID: 060174539dc945d5a28a2c71a54fae39
I20240501 02:07:41.981777 25054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:41.983060 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:41.973806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.22.12.126:41867)
0501 02:07:41.973994 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:41.974006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:41.974033 (+ 27us) server_negotiation.cc:407] Connection header received
0501 02:07:41.974450 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:41.974458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:41.974506 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:41.974661 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:41.975453 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:41.976227 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:41.976783 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:41.977013 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:41.979454 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:41.979465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:41.979465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:41.979490 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:41.981599 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:41.981976 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:41.981979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:41.981979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:41.982042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:41.982204 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:41.982204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:41.982205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:41.982433 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:41.982582 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:41.982730 (+ 148us) server_negotiation.cc:299] Negotiation successful
0501 02:07:41.982855 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20240501 02:07:41.989651 25054 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:41.990198 25054 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:41.998693 25054 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3dc0de62dcc141a6a16cde458fe09bdc: Generated new TSK 0
I20240501 02:07:41.998919 25054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:42.033828 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41867
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:42.125900 25066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:42.126088 25066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:42.126112 25066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:42.127506 25066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:42.127559 25066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:42.127586 25066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:42.129248 25066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:42.129460 25066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:42.129652 25066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:42.134513 25074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:42.134822 25072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:42.135301 25071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:42.135773 25066 server_base.cc:1034] running on GCE node
I20240501 02:07:42.135970 25066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:42.136186 25066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:42.137334 25066 hybrid_clock.cc:648] HybridClock initialized: now 1714529262137320 us; error 34 us; skew 500 ppm
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[24959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:42.140110 25066 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:42.140606 25066 webserver.cc:460] Webserver started at http://127.22.12.65:43885/ using document root <none> and password file <none>
I20240501 02:07:42.140812 25066 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:42.140856 25066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:42.140966 25066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:42.141953 25066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "6be6f2fa6f5e4a95b6154ed16ca2f80d"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "10a6878d75fb79547e90a3d7ff0014be"
server_key_iv: "2542fe2e40d65c8a1f204e4797d9a060"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.142267 25066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "6be6f2fa6f5e4a95b6154ed16ca2f80d"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "10a6878d75fb79547e90a3d7ff0014be"
server_key_iv: "2542fe2e40d65c8a1f204e4797d9a060"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.143507 25066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:42.144155 25081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.144344 25066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:42.144440 25066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "6be6f2fa6f5e4a95b6154ed16ca2f80d"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "10a6878d75fb79547e90a3d7ff0014be"
server_key_iv: "2542fe2e40d65c8a1f204e4797d9a060"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.144517 25066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:42.156914 25066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:42.158668 25066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:42.158767 25066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:42.158957 25066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:42.159248 25066 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:42.159281 25066 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.159303 25066 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:42.159325 25066 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.164676 25066 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38825
I20240501 02:07:42.164746 25194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38825 every 8 connection(s)
I20240501 02:07:42.165022 25066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:42.169088 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25066
I20240501 02:07:42.169202 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:42.169376 22577 external_mini_cluster.cc:1428] Setting key 3a8cada75fd1537e54ba89fdd52a3e94
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[24959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:42.174029 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:42.165759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:55679 (local address 127.22.12.126:41867)
0501 02:07:42.165824 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:42.165826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:42.165999 (+ 173us) server_negotiation.cc:407] Connection header received
0501 02:07:42.166438 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:42.166440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:42.166449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:42.166520 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:42.167386 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.167878 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.168462 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.168609 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.170974 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:42.170978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:42.170979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:42.170987 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:42.172488 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.172998 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.172999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:42.172999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:42.173042 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.173303 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.173304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:42.173304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:42.173404 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:42.173455 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:42.173846 (+ 391us) server_negotiation.cc:299] Negotiation successful
0501 02:07:42.173937 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 02:07:42.174144 25197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:42.165709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41867 (local address 127.22.12.65:55679)
0501 02:07:42.165881 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:42.165912 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:42.166356 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:42.166640 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:42.166643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:42.166992 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:42.167291 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:42.167295 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.167954 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.167955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:42.168362 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:42.168364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.168715 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.169377 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:42.169396 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:42.170876 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:42.172583 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.172591 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:42.172594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:42.172880 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.173122 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.173123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:42.173124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:42.173241 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.173461 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:42.173468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:42.173635 (+ 167us) client_negotiation.cc:769] Sending connection context
0501 02:07:42.173771 (+ 136us) client_negotiation.cc:240] Negotiation successful
0501 02:07:42.173939 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20240501 02:07:42.174778 25195 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41867
I20240501 02:07:42.174891 25195 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:42.175138 25195 heartbeater.cc:511] Master 127.22.12.126:41867 requested a full tablet report, sending...
I20240501 02:07:42.175714 25004 ts_manager.cc:194] Registered new tserver with Master: 6be6f2fa6f5e4a95b6154ed16ca2f80d (127.22.12.65:38825)
I20240501 02:07:42.176390 25004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:55679
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:42.197461 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41867
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:42.276332 25200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:42.276544 25200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:42.276568 25200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:42.277997 25200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:42.278064 25200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:42.278090 25200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:42.280308 25200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:42.280565 25200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:42.280743 25200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:42.285382 25205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:42.285575 25208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:42.286070 25206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:42.286446 25200 server_base.cc:1034] running on GCE node
I20240501 02:07:42.286665 25200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:42.286998 25200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:42.288188 25200 hybrid_clock.cc:648] HybridClock initialized: now 1714529262288158 us; error 49 us; skew 500 ppm
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[24959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:42.290684 25200 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:42.291170 25200 webserver.cc:460] Webserver started at http://127.22.12.66:43985/ using document root <none> and password file <none>
I20240501 02:07:42.291347 25200 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:42.291388 25200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:42.291493 25200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:42.292479 25200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "28f4525b1c31449cb21e8183c23630ff"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "611514eb0b372b7a92b91cc06f133044"
server_key_iv: "b7bc277fd2f67ebb478bc59b5c2aa18b"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.292781 25200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "28f4525b1c31449cb21e8183c23630ff"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "611514eb0b372b7a92b91cc06f133044"
server_key_iv: "b7bc277fd2f67ebb478bc59b5c2aa18b"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.293943 25200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:42.294611 25215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.294768 25200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:42.294847 25200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "28f4525b1c31449cb21e8183c23630ff"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "611514eb0b372b7a92b91cc06f133044"
server_key_iv: "b7bc277fd2f67ebb478bc59b5c2aa18b"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.294909 25200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:42.320459 25200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:42.322515 25200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:42.322654 25200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:42.322855 25200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:42.323173 25200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:42.323206 25200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.323236 25200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:42.323252 25200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.328294 25200 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:34841
I20240501 02:07:42.328343 25328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:34841 every 8 connection(s)
I20240501 02:07:42.328624 25200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:42.332334 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25200
I20240501 02:07:42.332453 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:42.332616 22577 external_mini_cluster.cc:1428] Setting key 4b3f3ec1211d0150b89336ea45391a6e
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[24959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:42.337383 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:42.329318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:60119 (local address 127.22.12.126:41867)
0501 02:07:42.329397 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:42.329400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:42.329524 (+ 124us) server_negotiation.cc:407] Connection header received
0501 02:07:42.329984 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:42.329985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:42.329996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:42.330051 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:42.330835 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.331429 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.332020 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.332143 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.334620 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:42.334625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:42.334626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:42.334634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:42.336083 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.336546 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.336546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:42.336547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:42.336568 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.336763 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.336763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:42.336763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:42.336856 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:42.336897 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:42.337239 (+ 342us) server_negotiation.cc:299] Negotiation successful
0501 02:07:42.337311 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 02:07:42.337589 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:42.329244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41867 (local address 127.22.12.66:60119)
0501 02:07:42.329391 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:42.329418 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:42.329874 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:42.330110 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:42.330114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:42.330434 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:42.330748 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:42.330753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.331539 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.331540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:42.331928 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:42.331929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.332313 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.332895 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:42.332921 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:42.334469 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:42.336171 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.336177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:42.336181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:42.336472 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.336626 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.336627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:42.336627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:42.336708 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.336905 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:42.336909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:42.337102 (+ 193us) client_negotiation.cc:769] Sending connection context
0501 02:07:42.337235 (+ 133us) client_negotiation.cc:240] Negotiation successful
0501 02:07:42.337386 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20240501 02:07:42.338138 25329 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41867
I20240501 02:07:42.338251 25329 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:42.338485 25329 heartbeater.cc:511] Master 127.22.12.126:41867 requested a full tablet report, sending...
I20240501 02:07:42.338925 25004 ts_manager.cc:194] Registered new tserver with Master: 28f4525b1c31449cb21e8183c23630ff (127.22.12.66:34841)
I20240501 02:07:42.339300 25004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:60119
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:42.358160 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41867
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:42.436872 25334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:42.437062 25334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:42.437083 25334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:42.438421 25334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:42.438474 25334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:42.438499 25334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:42.440474 25334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:42.440718 25334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:42.440898 25334 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:42.445395 25342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:42.445648 25339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:42.446177 25340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:42.446549 25334 server_base.cc:1034] running on GCE node
I20240501 02:07:42.446717 25334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:42.446920 25334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:42.448083 25334 hybrid_clock.cc:648] HybridClock initialized: now 1714529262448064 us; error 28 us; skew 500 ppm
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[24959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:42.450451 25334 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:42.450909 25334 webserver.cc:460] Webserver started at http://127.22.12.67:40285/ using document root <none> and password file <none>
I20240501 02:07:42.451086 25334 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:42.451128 25334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:42.451238 25334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:42.452239 25334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "65bc83ad3f464652a166b32e591ff947"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "db4a030c400fa0b9da160e05668e619a"
server_key_iv: "1ee2c717b731b506ba3307f38bcb6f07"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.452543 25334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "65bc83ad3f464652a166b32e591ff947"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "db4a030c400fa0b9da160e05668e619a"
server_key_iv: "1ee2c717b731b506ba3307f38bcb6f07"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.453656 25334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:42.454274 25349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.454417 25334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:42.454489 25334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "65bc83ad3f464652a166b32e591ff947"
format_stamp: "Formatted at 2024-05-01 02:07:42 on dist-test-slave-9tkh"
server_key: "db4a030c400fa0b9da160e05668e619a"
server_key_iv: "1ee2c717b731b506ba3307f38bcb6f07"
server_key_version: "encryptionkey@0"
I20240501 02:07:42.454581 25334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:42.468948 25334 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:42.471099 25334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:42.471205 25334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:42.471413 25334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:42.471714 25334 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:42.471746 25334 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.471768 25334 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:42.471791 25334 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:42.477068 25334 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:40233
I20240501 02:07:42.477111 25462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:40233 every 8 connection(s)
I20240501 02:07:42.477416 25334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:42.482657 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25334
I20240501 02:07:42.482769 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:42.482983 22577 external_mini_cluster.cc:1428] Setting key f16029266a258a93f03c242f4ca44bb0
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[24959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:42.486048 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:42.478179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:36369 (local address 127.22.12.126:41867)
0501 02:07:42.478233 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:42.478235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:42.478347 (+ 112us) server_negotiation.cc:407] Connection header received
0501 02:07:42.478900 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:42.478902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:42.478913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:42.478981 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:42.479833 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.480406 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.481034 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.481199 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.483607 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:42.483611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:42.483611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:42.483618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:42.485014 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.485387 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.485388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:42.485388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:42.485406 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.485541 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.485542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:42.485542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:42.485631 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:42.485672 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:42.485892 (+ 220us) server_negotiation.cc:299] Negotiation successful
0501 02:07:42.485965 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:07:42.486158 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:42.478096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41867 (local address 127.22.12.67:36369)
0501 02:07:42.478243 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:42.478273 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:42.478759 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:42.479093 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:42.479097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:42.479414 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:42.479743 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:42.479747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.480491 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.480492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:42.480953 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:42.480955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.481341 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.481840 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:42.481854 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:42.483518 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:42.485121 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.485127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:42.485129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:42.485327 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.485452 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.485452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:42.485452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:42.485502 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.485671 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:42.485673 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:42.485784 (+ 111us) client_negotiation.cc:769] Sending connection context
0501 02:07:42.485865 (+ 81us) client_negotiation.cc:240] Negotiation successful
0501 02:07:42.485981 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20240501 02:07:42.486586 25463 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41867
I20240501 02:07:42.486677 25463 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:42.486850 25463 heartbeater.cc:511] Master 127.22.12.126:41867 requested a full tablet report, sending...
I20240501 02:07:42.487228 25004 ts_manager.cc:194] Registered new tserver with Master: 65bc83ad3f464652a166b32e591ff947 (127.22.12.67:40233)
I20240501 02:07:42.487623 25004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:36369
I20240501 02:07:42.495043 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240501 02:07:42.504709 25043 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:07:42.503878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.22.12.126:41867)
0501 02:07:42.503940 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:42.503942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:42.504019 (+ 77us) server_negotiation.cc:407] Connection header received
0501 02:07:42.504074 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:42.504074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:42.504084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:42.504133 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:42.504629 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49630: BlockingRecv error: recv got EOF from 127.0.0.1:49630 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:07:42.505497 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25066
I20240501 02:07:42.509711 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25200
I20240501 02:07:42.513593 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25334
I20240501 02:07:42.517338 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 24972
2024-05-01T02:07:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1679 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[25475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[25475](info): set up 2 sockets
May 01 02:07:42 dist-test-slave-9tkh krb5kdc[25475](info): commencing operation
krb5kdc: starting...
W20240501 02:07:43.414160 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:43 dist-test-slave-9tkh krb5kdc[25475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:43.525789 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:33273
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:33273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:43.606685 25488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:43.606881 25488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:43.606912 25488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:43.608234 25488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:43.608281 25488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:43.608305 25488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:43.608320 25488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:43.608335 25488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:43.609982 25488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:33273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:33273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:43.610213 25488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:43.610399 25488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:43.614631 25493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:43.615075 25496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:43.615427 25494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:43.615860 25488 server_base.cc:1034] running on GCE node
I20240501 02:07:43.616125 25488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:43.616434 25488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:43.617566 25488 hybrid_clock.cc:648] HybridClock initialized: now 1714529263617551 us; error 31 us; skew 500 ppm
May 01 02:07:43 dist-test-slave-9tkh krb5kdc[25475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529263, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:43.619953 25488 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:43.620375 25488 webserver.cc:460] Webserver started at http://127.22.12.126:35961/ using document root <none> and password file <none>
I20240501 02:07:43.620553 25488 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:43.620594 25488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:43.620697 25488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:43.621693 25488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17"
format_stamp: "Formatted at 2024-05-01 02:07:43 on dist-test-slave-9tkh"
server_key: "d7da76c0a594fe01b928c532c565c5b6"
server_key_iv: "e5fc2136942dac82ef087cb41a2bf7f9"
server_key_version: "encryptionkey@0"
I20240501 02:07:43.621980 25488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17"
format_stamp: "Formatted at 2024-05-01 02:07:43 on dist-test-slave-9tkh"
server_key: "d7da76c0a594fe01b928c532c565c5b6"
server_key_iv: "e5fc2136942dac82ef087cb41a2bf7f9"
server_key_version: "encryptionkey@0"
I20240501 02:07:43.623157 25488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:07:43.623780 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:43.623927 25488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:07:43.624006 25488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17"
format_stamp: "Formatted at 2024-05-01 02:07:43 on dist-test-slave-9tkh"
server_key: "d7da76c0a594fe01b928c532c565c5b6"
server_key_iv: "e5fc2136942dac82ef087cb41a2bf7f9"
server_key_version: "encryptionkey@0"
I20240501 02:07:43.624063 25488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:43.632500 25488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:43.634397 25488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:43.634517 25488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:43.638127 25488 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:33273
I20240501 02:07:43.638163 25555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:33273 every 8 connection(s)
I20240501 02:07:43.638463 25488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:43.639079 25556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:43.640180 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25488
I20240501 02:07:43.640273 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:43.640431 22577 external_mini_cluster.cc:1428] Setting key fdf05cea8fbed42b9302ef18ef4fef9c
I20240501 02:07:43.642362 25556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17: Bootstrap starting.
I20240501 02:07:43.643018 25556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:43.643270 25556 log.cc:824] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:43.643967 25556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17: No bootstrap required, opened a new log
May 01 02:07:43 dist-test-slave-9tkh krb5kdc[25475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:43.646008 25556 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33273 } }
I20240501 02:07:43.646173 25556 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:43.646209 25556 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e0a1e9616f4dcb9fbfc45b59e28b17, State: Initialized, Role: FOLLOWER
I20240501 02:07:43.646327 25556 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33273 } }
I20240501 02:07:43.646405 25556 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:43.646443 25556 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:43.646494 25556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:43.647204 25556 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33273 } }
I20240501 02:07:43.647325 25556 leader_election.cc:302] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e0a1e9616f4dcb9fbfc45b59e28b17; no voters:
I20240501 02:07:43.647496 25556 leader_election.cc:288] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:43.647562 25561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:43.647706 25561 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [term 1 LEADER]: Becoming Leader. State: Replica: f7e0a1e9616f4dcb9fbfc45b59e28b17, State: Running, Role: LEADER
I20240501 02:07:43.647730 25556 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:43.648476 25561 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33273 } }
I20240501 02:07:43.649032 25563 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e0a1e9616f4dcb9fbfc45b59e28b17. Latest consensus state: current_term: 1 leader_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33273 } } }
I20240501 02:07:43.649344 25563 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:43.649153 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e0a1e9616f4dcb9fbfc45b59e28b17" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33273 } } }
I20240501 02:07:43.649575 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17 [sys.catalog]: This master's current role is: LEADER
W20240501 02:07:43.649602 25575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:07:43.649691 25575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:07:43.649751 25578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:43.649928 25578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:43.650272 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:43.640911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.22.12.126:33273)
0501 02:07:43.641049 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:43.641058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:43.641076 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:07:43.641476 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:43.641481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:43.641526 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.641648 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:43.642622 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.643388 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.644052 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.644294 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.646520 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:43.646571 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:43.646571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:43.646594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:43.648526 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.648902 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.648904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:43.648905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:43.648970 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.649137 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.649139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:43.649139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:43.649401 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:43.649948 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:43.650034 (+ 86us) server_negotiation.cc:299] Negotiation successful
0501 02:07:43.650139 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20240501 02:07:43.651162 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:07:43.651271 25578 catalog_manager.cc:1347] Generated new cluster ID: c998b986e7a04d81901e07173cea2d9f
I20240501 02:07:43.651324 25578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:43.668812 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:43.665922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.22.12.126:33273)
0501 02:07:43.665967 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:43.665968 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:43.666090 (+ 122us) server_negotiation.cc:407] Connection header received
0501 02:07:43.666163 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:43.666164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:43.666175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.666221 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:43.666777 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.667234 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.667750 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.668216 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.668350 (+ 134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:43.668355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:43.668356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:43.668536 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:43.668550 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:43.668556 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:43.668609 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:43.668679 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 02:07:43.668719 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:07:43.678028 25578 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:43.678671 25578 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:43.683470 25578 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f7e0a1e9616f4dcb9fbfc45b59e28b17: Generated new TSK 0
I20240501 02:07:43.683681 25578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:07:43.691126 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25488
2024-05-01T02:07:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1172 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:43 dist-test-slave-9tkh krb5kdc[25589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:43 dist-test-slave-9tkh krb5kdc[25589](info): set up 2 sockets
May 01 02:07:43 dist-test-slave-9tkh krb5kdc[25589](info): commencing operation
krb5kdc: starting...
W20240501 02:07:44.582263 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:44 dist-test-slave-9tkh krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:44.697691 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39397
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:39397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:44.785832 25602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:44.786049 25602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:44.786072 25602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:44.787487 25602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:44.787536 25602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:44.787550 25602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:44.787560 25602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:44.787570 25602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:44.789125 25602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:39397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:44.789309 25602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:44.789459 25602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:44.793922 25608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:44.794415 25610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:44.794855 25607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:44.795300 25602 server_base.cc:1034] running on GCE node
I20240501 02:07:44.795524 25602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:44.795783 25602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:44.796913 25602 hybrid_clock.cc:648] HybridClock initialized: now 1714529264796900 us; error 38 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-9tkh krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:44.799400 25602 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:44.799808 25602 webserver.cc:460] Webserver started at http://127.22.12.126:36857/ using document root <none> and password file <none>
I20240501 02:07:44.799973 25602 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:44.800012 25602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:44.800122 25602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:44.801086 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "71e94a2c596247a492d46585757045af"
format_stamp: "Formatted at 2024-05-01 02:07:44 on dist-test-slave-9tkh"
server_key: "9047c1a31727538e84f00e34a95d25da"
server_key_iv: "479f89612a63ae540d58252b8663a744"
server_key_version: "encryptionkey@0"
I20240501 02:07:44.801390 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "71e94a2c596247a492d46585757045af"
format_stamp: "Formatted at 2024-05-01 02:07:44 on dist-test-slave-9tkh"
server_key: "9047c1a31727538e84f00e34a95d25da"
server_key_iv: "479f89612a63ae540d58252b8663a744"
server_key_version: "encryptionkey@0"
I20240501 02:07:44.802772 25602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:07:44.803503 25617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:44.803670 25602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:07:44.803757 25602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "71e94a2c596247a492d46585757045af"
format_stamp: "Formatted at 2024-05-01 02:07:44 on dist-test-slave-9tkh"
server_key: "9047c1a31727538e84f00e34a95d25da"
server_key_iv: "479f89612a63ae540d58252b8663a744"
server_key_version: "encryptionkey@0"
I20240501 02:07:44.803820 25602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:44.814749 25602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:44.816638 25602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:44.816778 25602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:44.820551 25602 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:39397
I20240501 02:07:44.820657 25669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:39397 every 8 connection(s)
I20240501 02:07:44.820933 25602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:44.821938 25670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:44.822285 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25602
I20240501 02:07:44.822381 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:44.822576 22577 external_mini_cluster.cc:1428] Setting key ba6deb893d0d79a4aeda241e83770ff0
I20240501 02:07:44.824774 25670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af: Bootstrap starting.
I20240501 02:07:44.825398 25670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:44.825678 25670 log.cc:824] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:44.826390 25670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af: No bootstrap required, opened a new log
I20240501 02:07:44.828133 25670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e94a2c596247a492d46585757045af" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39397 } }
I20240501 02:07:44.828267 25670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:44.828291 25670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e94a2c596247a492d46585757045af, State: Initialized, Role: FOLLOWER
May 01 02:07:44 dist-test-slave-9tkh krb5kdc[25589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:44.828392 25670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e94a2c596247a492d46585757045af" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39397 } }
I20240501 02:07:44.828457 25670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:44.828497 25670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:44.828543 25670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:44.829193 25670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e94a2c596247a492d46585757045af" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39397 } }
I20240501 02:07:44.829293 25670 leader_election.cc:302] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e94a2c596247a492d46585757045af; no voters:
I20240501 02:07:44.829478 25670 leader_election.cc:288] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:44.829541 25675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:44.829742 25670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:44.829758 25675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [term 1 LEADER]: Becoming Leader. State: Replica: 71e94a2c596247a492d46585757045af, State: Running, Role: LEADER
I20240501 02:07:44.830159 25675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e94a2c596247a492d46585757045af" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39397 } }
I20240501 02:07:44.830520 25677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e94a2c596247a492d46585757045af. Latest consensus state: current_term: 1 leader_uuid: "71e94a2c596247a492d46585757045af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e94a2c596247a492d46585757045af" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39397 } } }
I20240501 02:07:44.830623 25677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:44.830725 25676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e94a2c596247a492d46585757045af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e94a2c596247a492d46585757045af" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39397 } } }
I20240501 02:07:44.830773 25676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:44.830947 25684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:44.831511 25684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:44.832451 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:44.823154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.22.12.126:39397)
0501 02:07:44.823372 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:44.823386 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:44.823409 (+ 23us) server_negotiation.cc:407] Connection header received
0501 02:07:44.823903 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:44.823911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:44.823969 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.824142 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:44.824880 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.825965 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.826605 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.826836 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.828897 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:44.828906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:44.828906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:44.828929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:44.831100 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.831513 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.831515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:44.831515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:44.831563 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.831717 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.831718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:44.831718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:44.831916 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:44.832072 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:44.832167 (+ 95us) server_negotiation.cc:299] Negotiation successful
0501 02:07:44.832291 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20240501 02:07:44.835191 25684 catalog_manager.cc:1347] Generated new cluster ID: c2ea77505fbd418ea5b032a2b2d83de8
I20240501 02:07:44.835253 25684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:44.842562 25684 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:44.843407 25684 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:44.857129 25684 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 71e94a2c596247a492d46585757045af: Generated new TSK 0
I20240501 02:07:44.857358 25684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:44.885183 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39397
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:44.964838 25696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:44.965039 25696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:44.965068 25696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:44.966352 25696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:44.966396 25696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:44.966428 25696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:44.967918 25696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:44.968137 25696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:44.968312 25696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:44.973225 25704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:44.973655 25702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:44.974054 25701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:44.974522 25696 server_base.cc:1034] running on GCE node
I20240501 02:07:44.974768 25696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:44.974999 25696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:44.976159 25696 hybrid_clock.cc:648] HybridClock initialized: now 1714529264976144 us; error 32 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-9tkh krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:44.979153 25696 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:44.979736 25696 webserver.cc:460] Webserver started at http://127.22.12.65:44293/ using document root <none> and password file <none>
I20240501 02:07:44.979939 25696 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:44.979988 25696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:44.980099 25696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:44.981377 25696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "0e95b4ab6a6c45f5ac982e87067b5ac4"
format_stamp: "Formatted at 2024-05-01 02:07:44 on dist-test-slave-9tkh"
server_key: "8b2a3deb3c0841e9a4baca9887727140"
server_key_iv: "db0fd647d33a7ba245c7215626a678ea"
server_key_version: "encryptionkey@0"
I20240501 02:07:44.981760 25696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "0e95b4ab6a6c45f5ac982e87067b5ac4"
format_stamp: "Formatted at 2024-05-01 02:07:44 on dist-test-slave-9tkh"
server_key: "8b2a3deb3c0841e9a4baca9887727140"
server_key_iv: "db0fd647d33a7ba245c7215626a678ea"
server_key_version: "encryptionkey@0"
I20240501 02:07:44.983268 25696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:44.984079 25711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:44.984261 25696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:44.984344 25696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "0e95b4ab6a6c45f5ac982e87067b5ac4"
format_stamp: "Formatted at 2024-05-01 02:07:44 on dist-test-slave-9tkh"
server_key: "8b2a3deb3c0841e9a4baca9887727140"
server_key_iv: "db0fd647d33a7ba245c7215626a678ea"
server_key_version: "encryptionkey@0"
I20240501 02:07:44.984412 25696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:44.998755 25696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:45.000854 25696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:45.000959 25696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:45.001152 25696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:45.001401 25696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:45.001430 25696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.001451 25696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:45.001467 25696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.006803 25696 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:40425
I20240501 02:07:45.006892 25824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:40425 every 8 connection(s)
I20240501 02:07:45.007177 25696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:45.010037 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25696
I20240501 02:07:45.010146 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:45.010296 22577 external_mini_cluster.cc:1428] Setting key a10017c116226bc38e90e0b2ad585b6a
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:45.015784 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.007852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:54915 (local address 127.22.12.126:39397)
0501 02:07:45.007907 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:45.007910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:45.008092 (+ 182us) server_negotiation.cc:407] Connection header received
0501 02:07:45.008598 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:45.008599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:45.008611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:45.008667 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:45.009712 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.010228 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.010837 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.010993 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.013060 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:45.013066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:45.013066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:45.013073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:45.014626 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.014991 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.014992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.014993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.015017 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.015184 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.015185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.015185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.015283 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:45.015338 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:45.015581 (+ 243us) server_negotiation.cc:299] Negotiation successful
0501 02:07:45.015684 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:07:45.015930 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.007825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39397 (local address 127.22.12.65:54915)
0501 02:07:45.007953 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.007985 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:45.008487 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:45.008732 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:45.008736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:45.009041 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:45.009602 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:45.009606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.010310 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.010311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:45.010753 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:45.010756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.010894 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.011515 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:45.011534 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:45.012950 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:45.014713 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:45.014720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:45.014724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:45.014928 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:45.015068 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:45.015069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:45.015069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:45.015133 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:45.015333 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:45.015337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:45.015458 (+ 121us) client_negotiation.cc:769] Sending connection context
0501 02:07:45.015554 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 02:07:45.015689 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20240501 02:07:45.016476 25825 heartbeater.cc:347] Connected to a master server at 127.22.12.126:39397
I20240501 02:07:45.016592 25825 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:45.016836 25825 heartbeater.cc:511] Master 127.22.12.126:39397 requested a full tablet report, sending...
I20240501 02:07:45.017400 25633 ts_manager.cc:194] Registered new tserver with Master: 0e95b4ab6a6c45f5ac982e87067b5ac4 (127.22.12.65:40425)
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
I20240501 02:07:45.018044 25633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:54915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:45.034338 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39397
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:45.113618 25830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:45.113823 25830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:45.113854 25830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:45.115263 25830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:45.115312 25830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:45.115345 25830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:45.116870 25830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:45.117110 25830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:45.117291 25830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:45.121984 25836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:45.122124 25835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:45.122676 25838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:45.123140 25830 server_base.cc:1034] running on GCE node
I20240501 02:07:45.123315 25830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:45.123512 25830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:45.124637 25830 hybrid_clock.cc:648] HybridClock initialized: now 1714529265124621 us; error 39 us; skew 500 ppm
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:45.127125 25830 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:45.127566 25830 webserver.cc:460] Webserver started at http://127.22.12.66:40217/ using document root <none> and password file <none>
I20240501 02:07:45.127746 25830 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:45.127789 25830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:45.127899 25830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:45.128906 25830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "fcaaf9979cc445978a170edd72c76197"
format_stamp: "Formatted at 2024-05-01 02:07:45 on dist-test-slave-9tkh"
server_key: "a8eaad4cee3efb361d5dbf380a8276eb"
server_key_iv: "2b72b86037fb3b408d7c655d65d14e96"
server_key_version: "encryptionkey@0"
I20240501 02:07:45.129232 25830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "fcaaf9979cc445978a170edd72c76197"
format_stamp: "Formatted at 2024-05-01 02:07:45 on dist-test-slave-9tkh"
server_key: "a8eaad4cee3efb361d5dbf380a8276eb"
server_key_iv: "2b72b86037fb3b408d7c655d65d14e96"
server_key_version: "encryptionkey@0"
I20240501 02:07:45.130496 25830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:45.131279 25845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.131505 25830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:45.131604 25830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "fcaaf9979cc445978a170edd72c76197"
format_stamp: "Formatted at 2024-05-01 02:07:45 on dist-test-slave-9tkh"
server_key: "a8eaad4cee3efb361d5dbf380a8276eb"
server_key_iv: "2b72b86037fb3b408d7c655d65d14e96"
server_key_version: "encryptionkey@0"
I20240501 02:07:45.131668 25830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:45.149446 25830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:45.151523 25830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:45.151639 25830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:45.151854 25830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:45.152168 25830 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:45.152200 25830 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.152230 25830 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:45.152246 25830 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.157429 25830 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:33347
I20240501 02:07:45.157474 25958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:33347 every 8 connection(s)
I20240501 02:07:45.157781 25830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:45.158929 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25830
I20240501 02:07:45.159021 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:45.159178 22577 external_mini_cluster.cc:1428] Setting key 82c08766c414d11c3777951220a85cc1
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:45.166543 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.158416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:39399 (local address 127.22.12.126:39397)
0501 02:07:45.158464 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:45.158467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:45.158659 (+ 192us) server_negotiation.cc:407] Connection header received
0501 02:07:45.159160 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:45.159161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:45.159173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:45.159228 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:45.160444 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.160924 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.161628 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.161763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.163972 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:45.163976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:45.163977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:45.163983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:45.165277 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.165726 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.165727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.165727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.165744 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.165931 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.165931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.165931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.166018 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:45.166076 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:45.166390 (+ 314us) server_negotiation.cc:299] Negotiation successful
0501 02:07:45.166456 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:07:45.166785 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.158395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39397 (local address 127.22.12.66:39399)
0501 02:07:45.158524 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.158572 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:45.159052 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:45.159300 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:45.159305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:45.159718 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:45.160344 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:45.160350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.161005 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.161006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:45.161540 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:45.161543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.161682 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.162167 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:45.162189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:45.163875 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:45.165361 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:45.165368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:45.165372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:45.165657 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:45.165798 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:45.165798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:45.165799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:45.165879 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:45.166088 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:45.166094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:45.166258 (+ 164us) client_negotiation.cc:769] Sending connection context
0501 02:07:45.166388 (+ 130us) client_negotiation.cc:240] Negotiation successful
0501 02:07:45.166578 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
I20240501 02:07:45.167346 25959 heartbeater.cc:347] Connected to a master server at 127.22.12.126:39397
I20240501 02:07:45.167433 25959 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:45.167618 25959 heartbeater.cc:511] Master 127.22.12.126:39397 requested a full tablet report, sending...
I20240501 02:07:45.168010 25633 ts_manager.cc:194] Registered new tserver with Master: fcaaf9979cc445978a170edd72c76197 (127.22.12.66:33347)
I20240501 02:07:45.168406 25633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:39399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:45.185967 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39397
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:45.270061 25964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:45.270270 25964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:45.270300 25964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:45.271682 25964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:45.271736 25964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:45.271768 25964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:45.273299 25964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:39397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:45.273535 25964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:45.273723 25964 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:45.278241 25969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:45.278728 25970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:45.279109 25972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:45.279580 25964 server_base.cc:1034] running on GCE node
I20240501 02:07:45.279784 25964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:45.280002 25964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:45.281136 25964 hybrid_clock.cc:648] HybridClock initialized: now 1714529265281125 us; error 34 us; skew 500 ppm
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:45.283685 25964 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:45.284090 25964 webserver.cc:460] Webserver started at http://127.22.12.67:35605/ using document root <none> and password file <none>
I20240501 02:07:45.284256 25964 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:45.284297 25964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:45.284399 25964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:45.285377 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "43aa1a1bcc8a4e6db627022e270deea3"
format_stamp: "Formatted at 2024-05-01 02:07:45 on dist-test-slave-9tkh"
server_key: "e1107cedfbb600a728dff596dbd9cf3c"
server_key_iv: "53fca2253b6802456add5961dda3ff4f"
server_key_version: "encryptionkey@0"
I20240501 02:07:45.285681 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "43aa1a1bcc8a4e6db627022e270deea3"
format_stamp: "Formatted at 2024-05-01 02:07:45 on dist-test-slave-9tkh"
server_key: "e1107cedfbb600a728dff596dbd9cf3c"
server_key_iv: "53fca2253b6802456add5961dda3ff4f"
server_key_version: "encryptionkey@0"
I20240501 02:07:45.286913 25964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:45.287611 25979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.287818 25964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:45.287902 25964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "43aa1a1bcc8a4e6db627022e270deea3"
format_stamp: "Formatted at 2024-05-01 02:07:45 on dist-test-slave-9tkh"
server_key: "e1107cedfbb600a728dff596dbd9cf3c"
server_key_iv: "53fca2253b6802456add5961dda3ff4f"
server_key_version: "encryptionkey@0"
I20240501 02:07:45.287986 25964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:45.298097 25964 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:45.300244 25964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:45.300364 25964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:45.300570 25964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:45.300853 25964 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:45.300887 25964 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.300915 25964 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:45.300947 25964 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:45.306351 25964 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:38617
I20240501 02:07:45.306398 26092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:38617 every 8 connection(s)
I20240501 02:07:45.306759 25964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:45.310767 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 25964
I20240501 02:07:45.310864 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:45.311004 22577 external_mini_cluster.cc:1428] Setting key cb3a56c7d19c2a8d02f5dfbcf1f3e516
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:45.315204 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.308134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:56937 (local address 127.22.12.126:39397)
0501 02:07:45.308192 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:45.308194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:45.308205 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:07:45.308317 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:45.308318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:45.308327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:45.308355 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:45.309435 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.309916 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.310493 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.310677 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.312746 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:45.312749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:45.312749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:45.312755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:45.314021 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.314396 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.314397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.314398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.314421 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.314626 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.314627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.314627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.314721 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:45.314776 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:45.315024 (+ 248us) server_negotiation.cc:299] Negotiation successful
0501 02:07:45.315113 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:07:45.315321 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.307495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39397 (local address 127.22.12.67:56937)
0501 02:07:45.307697 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.307730 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:45.308212 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:45.308427 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:45.308431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:45.308750 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:45.309349 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:45.309354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.310023 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.310024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:45.310424 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:45.310426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.310596 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.311148 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:45.311177 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:45.312654 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:45.314117 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:45.314123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:45.314126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:45.314328 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:45.314473 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:45.314473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:45.314474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:45.314570 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:45.314773 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:45.314778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:45.314901 (+ 123us) client_negotiation.cc:769] Sending connection context
0501 02:07:45.314996 (+ 95us) client_negotiation.cc:240] Negotiation successful
0501 02:07:45.315136 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":128,"threads_started":1}
I20240501 02:07:45.315796 26093 heartbeater.cc:347] Connected to a master server at 127.22.12.126:39397
I20240501 02:07:45.315888 26093 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:45.316095 26093 heartbeater.cc:511] Master 127.22.12.126:39397 requested a full tablet report, sending...
I20240501 02:07:45.316462 25633 ts_manager.cc:194] Registered new tserver with Master: 43aa1a1bcc8a4e6db627022e270deea3 (127.22.12.67:38617)
I20240501 02:07:45.316844 25633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:56937
I20240501 02:07:45.322829 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529265, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[25589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529265, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:45.343603 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:45.337271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.22.12.126:39397)
0501 02:07:45.337357 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:45.337359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:45.337443 (+ 84us) server_negotiation.cc:407] Connection header received
0501 02:07:45.337518 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:45.337519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:45.337529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:45.337576 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:45.338264 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.338800 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:45.339352 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:45.339508 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:45.341427 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:45.341431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:45.341431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:45.341439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:45.342799 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.343086 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.343087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.343087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.343105 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:45.343204 (+ 99us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:45.343204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:45.343204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:45.343318 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:45.343392 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:45.343465 (+ 73us) server_negotiation.cc:299] Negotiation successful
0501 02:07:45.343527 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20240501 02:07:45.343828 25633 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43960
I20240501 02:07:45.344236 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25696
I20240501 02:07:45.348196 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25830
I20240501 02:07:45.352476 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25964
I20240501 02:07:45.356456 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 25602
2024-05-01T02:07:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1666 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[26105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[26105](info): set up 2 sockets
May 01 02:07:45 dist-test-slave-9tkh krb5kdc[26105](info): commencing operation
krb5kdc: starting...
W20240501 02:07:46.250389 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.875s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:46.371348 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:37383
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:46.454038 26118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:46.454252 26118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:46.454274 26118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:46.455713 26118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:46.455767 26118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:46.455781 26118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:46.455791 26118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:46.455803 26118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:46.457554 26118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:46.457787 26118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:46.457949 26118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:46.462389 26124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.462881 26126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.463362 26123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:46.463876 26118 server_base.cc:1034] running on GCE node
I20240501 02:07:46.464138 26118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:46.464392 26118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:46.465536 26118 hybrid_clock.cc:648] HybridClock initialized: now 1714529266465510 us; error 50 us; skew 500 ppm
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:46.468240 26118 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:46.468729 26118 webserver.cc:460] Webserver started at http://127.22.12.126:34951/ using document root <none> and password file <none>
I20240501 02:07:46.468912 26118 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:46.468956 26118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:46.469079 26118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:46.470176 26118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "6ba64e609c264bceae4344080fee758d"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "dbf8f3072df02f821067f2a3c9986f6b"
server_key_iv: "9bc89774b9a5108a0b394594a9d44cdf"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.470484 26118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "6ba64e609c264bceae4344080fee758d"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "dbf8f3072df02f821067f2a3c9986f6b"
server_key_iv: "9bc89774b9a5108a0b394594a9d44cdf"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.471787 26118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:46.472606 26133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.472908 26118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:46.473001 26118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "6ba64e609c264bceae4344080fee758d"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "dbf8f3072df02f821067f2a3c9986f6b"
server_key_iv: "9bc89774b9a5108a0b394594a9d44cdf"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.473074 26118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:46.490691 26118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:46.493055 26118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:46.493194 26118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:46.497161 26118 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:37383
I20240501 02:07:46.497202 26185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:37383 every 8 connection(s)
I20240501 02:07:46.497510 26118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:46.498369 26186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:46.500591 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d: Bootstrap starting.
I20240501 02:07:46.501181 26186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:46.501427 26186 log.cc:824] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:46.502202 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d: No bootstrap required, opened a new log
I20240501 02:07:46.503711 26186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba64e609c264bceae4344080fee758d" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37383 } }
I20240501 02:07:46.503845 26186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:46.503861 26186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba64e609c264bceae4344080fee758d, State: Initialized, Role: FOLLOWER
I20240501 02:07:46.503981 26186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba64e609c264bceae4344080fee758d" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37383 } }
I20240501 02:07:46.504112 26186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:46.504225 26186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:46.504278 26186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:46.504870 26186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba64e609c264bceae4344080fee758d" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37383 } }
I20240501 02:07:46.504971 26186 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba64e609c264bceae4344080fee758d; no voters:
I20240501 02:07:46.505136 26186 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:46.505254 26189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:46.505385 26186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:46.505414 26189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [term 1 LEADER]: Becoming Leader. State: Replica: 6ba64e609c264bceae4344080fee758d, State: Running, Role: LEADER
I20240501 02:07:46.505563 26189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba64e609c264bceae4344080fee758d" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37383 } }
I20240501 02:07:46.505990 26190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba64e609c264bceae4344080fee758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba64e609c264bceae4344080fee758d" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37383 } } }
I20240501 02:07:46.506155 26190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:46.506327 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26118
I20240501 02:07:46.506413 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:46.506604 22577 external_mini_cluster.cc:1428] Setting key f1d2d92d07da05a83a4dd889e3b24541
I20240501 02:07:46.506608 26198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:46.506062 26191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba64e609c264bceae4344080fee758d. Latest consensus state: current_term: 1 leader_uuid: "6ba64e609c264bceae4344080fee758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba64e609c264bceae4344080fee758d" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37383 } } }
I20240501 02:07:46.506943 26191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:46.507061 26198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:46.511018 26198 catalog_manager.cc:1347] Generated new cluster ID: 263a5189293744958825cfe55322b398
I20240501 02:07:46.511078 26198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:46.520648 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:46.507221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.22.12.126:37383)
0501 02:07:46.510730 (+ 3509us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:46.510742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:46.510773 (+ 31us) server_negotiation.cc:407] Connection header received
0501 02:07:46.510887 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:46.510893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:46.510947 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:46.511126 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:46.512056 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.513335 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:46.514096 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.514363 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:46.517086 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:46.517094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:46.517095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:46.517126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:46.519213 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:46.519601 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:46.519604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:46.519604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:46.519660 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:46.519803 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:46.519804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:46.519804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:46.520044 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:46.520183 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:46.520318 (+ 135us) server_negotiation.cc:299] Negotiation successful
0501 02:07:46.520444 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3457,"thread_start_us":67,"threads_started":1}
I20240501 02:07:46.525727 26198 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:46.526281 26198 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:46.529793 26198 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6ba64e609c264bceae4344080fee758d: Generated new TSK 0
I20240501 02:07:46.529986 26198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:46.557894 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:37383
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:46.640398 26212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:46.640600 26212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:46.640631 26212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:46.641988 26212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:46.642040 26212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:46.642072 26212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:46.643843 26212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:46.644093 26212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:46.644282 26212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:46.648699 26218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.649127 26217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.649561 26220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:46.650013 26212 server_base.cc:1034] running on GCE node
I20240501 02:07:46.650231 26212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:46.650457 26212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:46.651620 26212 hybrid_clock.cc:648] HybridClock initialized: now 1714529266651605 us; error 34 us; skew 500 ppm
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:46.654304 26212 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:46.654774 26212 webserver.cc:460] Webserver started at http://127.22.12.65:44741/ using document root <none> and password file <none>
I20240501 02:07:46.654944 26212 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:46.654984 26212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:46.655090 26212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:46.656077 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "b8243842314e43f7bd8ef2e745b6dfd6"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "03f0f6668a3f12e797d0917d8d338797"
server_key_iv: "603daf9598f517c1fc472f23f4df0a45"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.656378 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "b8243842314e43f7bd8ef2e745b6dfd6"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "03f0f6668a3f12e797d0917d8d338797"
server_key_iv: "603daf9598f517c1fc472f23f4df0a45"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.657554 26212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:46.658267 26227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.658577 26212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:46.658805 26212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "b8243842314e43f7bd8ef2e745b6dfd6"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "03f0f6668a3f12e797d0917d8d338797"
server_key_iv: "603daf9598f517c1fc472f23f4df0a45"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.658881 26212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:46.705307 26212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:46.707657 26212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:46.707772 26212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:46.707983 26212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:46.708298 26212 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:46.708333 26212 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.708364 26212 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:46.708381 26212 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.713791 26212 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:44783
I20240501 02:07:46.713855 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:44783 every 8 connection(s)
I20240501 02:07:46.714134 26212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:46.722795 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:46.714943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:57825 (local address 127.22.12.126:37383)
0501 02:07:46.714998 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:46.715000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:46.715182 (+ 182us) server_negotiation.cc:407] Connection header received
0501 02:07:46.715649 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:46.715650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:46.715678 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:46.715751 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:46.716650 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.717183 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:46.717765 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.717905 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:46.720276 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:46.720282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:46.720282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:46.720290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:46.721627 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:46.722023 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:46.722024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:46.722024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:46.722046 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:46.722201 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:46.722201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:46.722202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:46.722308 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:46.722352 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:46.722632 (+ 280us) server_negotiation.cc:299] Negotiation successful
0501 02:07:46.722717 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:07:46.722911 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:46.714891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:37383 (local address 127.22.12.65:57825)
0501 02:07:46.715043 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:46.715073 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:46.715518 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:46.715863 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:46.715867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:46.716234 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:46.716531 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:46.716535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.717275 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:46.717277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:46.717664 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:46.717666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.718026 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:46.718647 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:46.718665 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:46.720156 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:46.721732 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:46.721738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:46.721741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:46.721956 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:46.722101 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:46.722101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:46.722102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:46.722156 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:46.722351 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:46.722354 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:46.722471 (+ 117us) client_negotiation.cc:769] Sending connection context
0501 02:07:46.722622 (+ 151us) client_negotiation.cc:240] Negotiation successful
0501 02:07:46.722744 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20240501 02:07:46.723495 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26212
I20240501 02:07:46.723546 26341 heartbeater.cc:347] Connected to a master server at 127.22.12.126:37383
I20240501 02:07:46.723623 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:46.723642 26341 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:46.723804 22577 external_mini_cluster.cc:1428] Setting key 29dadc4ca01538cdbdfabb57a719adbd
I20240501 02:07:46.723883 26341 heartbeater.cc:511] Master 127.22.12.126:37383 requested a full tablet report, sending...
I20240501 02:07:46.724409 26149 ts_manager.cc:194] Registered new tserver with Master: b8243842314e43f7bd8ef2e745b6dfd6 (127.22.12.65:44783)
I20240501 02:07:46.725129 26149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:57825
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:46.750371 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:37383
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:46.829422 26346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:46.829619 26346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:46.829654 26346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:46.831056 26346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:46.831110 26346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:46.831146 26346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:46.832849 26346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:46.833107 26346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:46.833294 26346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:46.837711 26354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.838069 26351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.838513 26352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:46.838961 26346 server_base.cc:1034] running on GCE node
I20240501 02:07:46.839195 26346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:46.839419 26346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:46.840561 26346 hybrid_clock.cc:648] HybridClock initialized: now 1714529266840544 us; error 39 us; skew 500 ppm
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:46.843025 26346 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:46.843497 26346 webserver.cc:460] Webserver started at http://127.22.12.66:43559/ using document root <none> and password file <none>
I20240501 02:07:46.843679 26346 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:46.843722 26346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:46.843844 26346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:46.844827 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "a52afc0bd8e1437e9fca3e5bfa6ab6ea"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "61d1121d867c66192932b749c72a4464"
server_key_iv: "470225baebf9311f3d6046090a4d5749"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.845122 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "a52afc0bd8e1437e9fca3e5bfa6ab6ea"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "61d1121d867c66192932b749c72a4464"
server_key_iv: "470225baebf9311f3d6046090a4d5749"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.846275 26346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:46.846992 26361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.847184 26346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:07:46.847268 26346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "a52afc0bd8e1437e9fca3e5bfa6ab6ea"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "61d1121d867c66192932b749c72a4464"
server_key_iv: "470225baebf9311f3d6046090a4d5749"
server_key_version: "encryptionkey@0"
I20240501 02:07:46.847326 26346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:46.858459 26346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:46.860423 26346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:46.860524 26346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:46.860680 26346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:46.861003 26346 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:46.861037 26346 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.861066 26346 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:46.861115 26346 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:46.866379 26346 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:40545
I20240501 02:07:46.866436 26474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:40545 every 8 connection(s)
I20240501 02:07:46.866744 26346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:46.875047 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26346
I20240501 02:07:46.875164 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:46.875342 22577 external_mini_cluster.cc:1428] Setting key 4bfb3837ac564c3303189d63ed006e4e
I20240501 02:07:46.875720 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:46.867423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:49817 (local address 127.22.12.126:37383)
0501 02:07:46.867470 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:46.867472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:46.867654 (+ 182us) server_negotiation.cc:407] Connection header received
0501 02:07:46.868114 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:46.868115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:46.868124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:46.868177 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:46.868919 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.869452 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:46.870105 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.870247 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:46.872580 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:46.872587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:46.872587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:46.872594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:46.874122 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:46.874518 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:46.874519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:46.874519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:46.874588 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:46.874782 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:46.874784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:46.874784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:46.874985 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:46.875057 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:46.875477 (+ 420us) server_negotiation.cc:299] Negotiation successful
0501 02:07:46.875585 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:07:46.875818 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:46.867403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:37383 (local address 127.22.12.66:49817)
0501 02:07:46.867548 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:46.867573 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:46.868013 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:46.868233 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:46.868237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:46.868530 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:46.868825 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:46.868829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.869572 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:46.869574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:46.869988 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:46.869989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:46.870351 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:46.870991 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:46.871009 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:46.872461 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:46.874227 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:46.874234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:46.874237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:46.874448 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:46.874636 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:46.874637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:46.874637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:46.874704 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:46.875077 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:46.875085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:46.875298 (+ 213us) client_negotiation.cc:769] Sending connection context
0501 02:07:46.875441 (+ 143us) client_negotiation.cc:240] Negotiation successful
0501 02:07:46.875614 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20240501 02:07:46.876468 26475 heartbeater.cc:347] Connected to a master server at 127.22.12.126:37383
I20240501 02:07:46.876591 26475 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:46.876817 26475 heartbeater.cc:511] Master 127.22.12.126:37383 requested a full tablet report, sending...
I20240501 02:07:46.877236 26149 ts_manager.cc:194] Registered new tserver with Master: a52afc0bd8e1437e9fca3e5bfa6ab6ea (127.22.12.66:40545)
I20240501 02:07:46.877687 26149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:49817
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:46.902390 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:37383
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:46.982113 26480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:46.982326 26480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:46.982360 26480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:46.983808 26480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:46.983865 26480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:46.983901 26480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:46.985612 26480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:46.985859 26480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:46.986063 26480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 02:07:46.991456 26480 server_base.cc:1034] running on GCE node
W20240501 02:07:46.991905 26485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.992568 26486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:46.992810 26488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:46.993006 26480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:46.993271 26480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:46.994405 26480 hybrid_clock.cc:648] HybridClock initialized: now 1714529266994388 us; error 35 us; skew 500 ppm
May 01 02:07:46 dist-test-slave-9tkh krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:47.001814 26480 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:47.002295 26480 webserver.cc:460] Webserver started at http://127.22.12.67:44495/ using document root <none> and password file <none>
I20240501 02:07:47.002467 26480 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:47.002509 26480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:47.002655 26480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:47.003644 26480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "84a434344a364dfc9f0211653da63f61"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "04eece61105b263bfed8b3b0f65d06cd"
server_key_iv: "f7babd9342f6ba9306094be3223ddc17"
server_key_version: "encryptionkey@0"
I20240501 02:07:47.003934 26480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "84a434344a364dfc9f0211653da63f61"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "04eece61105b263bfed8b3b0f65d06cd"
server_key_iv: "f7babd9342f6ba9306094be3223ddc17"
server_key_version: "encryptionkey@0"
I20240501 02:07:47.005141 26480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:47.005811 26495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:47.006000 26480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:47.006078 26480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "84a434344a364dfc9f0211653da63f61"
format_stamp: "Formatted at 2024-05-01 02:07:46 on dist-test-slave-9tkh"
server_key: "04eece61105b263bfed8b3b0f65d06cd"
server_key_iv: "f7babd9342f6ba9306094be3223ddc17"
server_key_version: "encryptionkey@0"
I20240501 02:07:47.006146 26480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:47.016323 26480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:47.018378 26480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:47.018502 26480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:47.018785 26480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:47.019078 26480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:47.019109 26480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:47.019136 26480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:47.019165 26480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:47.024542 26480 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:38647
I20240501 02:07:47.024581 26608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:38647 every 8 connection(s)
I20240501 02:07:47.024868 26480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:47.026957 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26480
I20240501 02:07:47.027040 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:47.027199 22577 external_mini_cluster.cc:1428] Setting key 2ec4e44b3a710c11d4f2999adc772ce7
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529266, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:47.033893 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:47.025489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:53065 (local address 127.22.12.126:37383)
0501 02:07:47.025529 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:47.025530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:47.025709 (+ 179us) server_negotiation.cc:407] Connection header received
0501 02:07:47.026157 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:47.026158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:47.026168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:47.026218 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:47.026997 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.027665 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:47.028321 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.028448 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:47.030672 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:47.030676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:47.030677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:47.030684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:47.032283 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:47.032934 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:47.032935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:47.032936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:47.032964 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:47.033235 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:47.033236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:47.033236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:47.033339 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:47.033397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:47.033733 (+ 336us) server_negotiation.cc:299] Negotiation successful
0501 02:07:47.033822 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:07:47.033995 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:47.025475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:37383 (local address 127.22.12.67:53065)
0501 02:07:47.025598 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:47.025623 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:47.026061 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:47.026265 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:47.026268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:47.026617 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:47.026900 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:47.026904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.027749 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:47.027750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:47.028248 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:47.028250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.028586 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:47.029058 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:47.029072 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:47.030515 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:47.032370 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:47.032377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:47.032381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:47.032840 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:47.033077 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:47.033078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:47.033078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:47.033151 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:47.033449 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:47.033453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:47.033588 (+ 135us) client_negotiation.cc:769] Sending connection context
0501 02:07:47.033695 (+ 107us) client_negotiation.cc:240] Negotiation successful
0501 02:07:47.033819 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
I20240501 02:07:47.034461 26609 heartbeater.cc:347] Connected to a master server at 127.22.12.126:37383
I20240501 02:07:47.034579 26609 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:47.034842 26609 heartbeater.cc:511] Master 127.22.12.126:37383 requested a full tablet report, sending...
I20240501 02:07:47.035351 26149 ts_manager.cc:194] Registered new tserver with Master: 84a434344a364dfc9f0211653da63f61 (127.22.12.67:38647)
I20240501 02:07:47.035719 26149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:53065
I20240501 02:07:47.039085 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:07:47.060389 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:47.053129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.22.12.65:44783)
0501 02:07:47.053304 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:47.053311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:47.053321 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:47.053347 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:47.053349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:47.053361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:47.053428 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:47.054076 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.054808 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:47.055454 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.055671 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:47.057592 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:47.057609 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:47.057610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:47.057617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:47.059500 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:47.059830 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:47.059833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:47.059834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:47.059868 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:47.059972 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:47.059973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:47.059973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:47.060066 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:47.060165 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:47.060225 (+ 60us) server_negotiation.cc:299] Negotiation successful
0501 02:07:47.060295 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:47.067672 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:47.061729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.22.12.126:37383)
0501 02:07:47.061790 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:47.061792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:47.061840 (+ 48us) server_negotiation.cc:407] Connection header received
0501 02:07:47.061894 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:47.061894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:47.061903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:47.061945 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:47.062560 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.063060 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:47.063543 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:47.063665 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:47.065551 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:47.065556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:47.065556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:47.065564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:47.066899 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:47.067174 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:47.067175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:47.067175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:47.067192 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:47.067283 (+ 91us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:47.067283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:47.067283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:47.067393 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:47.067463 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:47.067536 (+ 73us) server_negotiation.cc:299] Negotiation successful
0501 02:07:47.067597 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20240501 02:07:47.067891 26149 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47858
I20240501 02:07:47.068200 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26212
I20240501 02:07:47.073235 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26346
I20240501 02:07:47.077431 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26480
I20240501 02:07:47.081224 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26118
2024-05-01T02:07:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1725 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26621](info): set up 2 sockets
May 01 02:07:47 dist-test-slave-9tkh krb5kdc[26621](info): commencing operation
krb5kdc: starting...
W20240501 02:07:47.986435 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.885s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:07:48.106760 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41583
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:48.190279 26634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:48.190498 26634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:48.190567 26634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:48.192778 26634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:48.192865 26634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:48.192886 26634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:48.192905 26634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:48.192919 26634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:48.194962 26634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:48.195252 26634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:48.195456 26634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:48.200371 26642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.200583 26640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.201206 26639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:48.202298 26634 server_base.cc:1034] running on GCE node
I20240501 02:07:48.202571 26634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:48.202847 26634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:48.204023 26634 hybrid_clock.cc:648] HybridClock initialized: now 1714529268204012 us; error 42 us; skew 500 ppm
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:48.206621 26634 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:07:48.207104 26634 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:48.207165 26634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:48.207268 26634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:48.208307 26634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "3cf097952abeba276bd1b97033e532ea"
server_key_iv: "051fd39a9420eac1a8b9db626912266d"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.208635 26634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "3cf097952abeba276bd1b97033e532ea"
server_key_iv: "051fd39a9420eac1a8b9db626912266d"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.209878 26634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:07:48.210727 26648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.210987 26634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:48.211078 26634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "3cf097952abeba276bd1b97033e532ea"
server_key_iv: "051fd39a9420eac1a8b9db626912266d"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.211160 26634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:48.230670 26634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:48.232857 26634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:48.232987 26634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:48.237373 26634 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41583
I20240501 02:07:48.237407 26700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41583 every 8 connection(s)
I20240501 02:07:48.237672 26634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:48.238252 26701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:48.240913 26701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9: Bootstrap starting.
I20240501 02:07:48.241562 26701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:48.241832 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26634
I20240501 02:07:48.241858 26701 log.cc:824] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:48.241931 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:48.242079 22577 external_mini_cluster.cc:1428] Setting key 16dabdbf0094900d41fb935a19cf18c0
I20240501 02:07:48.242741 26701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9: No bootstrap required, opened a new log
I20240501 02:07:48.244381 26701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41583 } }
I20240501 02:07:48.244540 26701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:48.244567 26701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd19ae9626b43f1ad15c4c1f5bd96c9, State: Initialized, Role: FOLLOWER
I20240501 02:07:48.244721 26701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41583 } }
I20240501 02:07:48.244800 26701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:48.244822 26701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:48.244855 26701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:48.245460 26701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41583 } }
I20240501 02:07:48.245572 26701 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd19ae9626b43f1ad15c4c1f5bd96c9; no voters:
I20240501 02:07:48.245754 26701 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:48.245806 26706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:48.245997 26706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd19ae9626b43f1ad15c4c1f5bd96c9, State: Running, Role: LEADER
I20240501 02:07:48.246026 26701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:48.246090 26706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41583 } }
I20240501 02:07:48.246438 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd19ae9626b43f1ad15c4c1f5bd96c9. Latest consensus state: current_term: 1 leader_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41583 } } }
I20240501 02:07:48.246446 26707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd19ae9626b43f1ad15c4c1f5bd96c9" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41583 } } }
I20240501 02:07:48.246755 26707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:48.246940 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:48.247246 26715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:48.247761 26715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:48.249778 26715 catalog_manager.cc:1347] Generated new cluster ID: 500cfedfda0f4ab68468d30c96952e28
I20240501 02:07:48.249832 26715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:48.253974 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.242664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.22.12.126:41583)
0501 02:07:48.242868 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.242882 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.242944 (+ 62us) server_negotiation.cc:407] Connection header received
0501 02:07:48.244352 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.244362 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.244438 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.244599 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:48.245418 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.247024 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.248065 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.248327 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.250433 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:48.250454 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:48.250454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:48.250473 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:48.252584 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.252983 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.252985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.252985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.253039 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.253181 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.253182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.253182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.253398 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:48.253548 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.253658 (+ 110us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.253781 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20240501 02:07:48.260902 26715 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:48.261452 26715 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:48.264582 26715 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9: Generated new TSK 0
I20240501 02:07:48.264747 26715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:07:48.294960 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41583
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:48.376425 26727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:48.376633 26727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:48.376670 26727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:48.378283 26727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:48.378360 26727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:48.378401 26727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:48.380565 26727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:48.380841 26727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:48.381035 26727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:48.385681 26732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.386066 26733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.386584 26735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:48.387102 26727 server_base.cc:1034] running on GCE node
I20240501 02:07:48.387328 26727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:48.387637 26727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:48.388797 26727 hybrid_clock.cc:648] HybridClock initialized: now 1714529268388781 us; error 36 us; skew 500 ppm
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:48.391971 26727 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:07:48.392343 26727 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:48.392403 26727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:48.392506 26727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:48.393841 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "806abf8826a94f8b8d1b0cbd57dfcac5"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "7eada435d7998b59c2a59c346dfd0bf0"
server_key_iv: "d840cf332eadb2d2cecfbc5d2a404532"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.394191 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "806abf8826a94f8b8d1b0cbd57dfcac5"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "7eada435d7998b59c2a59c346dfd0bf0"
server_key_iv: "d840cf332eadb2d2cecfbc5d2a404532"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.395745 26727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:07:48.397348 26741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.397549 26727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:07:48.397692 26727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "806abf8826a94f8b8d1b0cbd57dfcac5"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "7eada435d7998b59c2a59c346dfd0bf0"
server_key_iv: "d840cf332eadb2d2cecfbc5d2a404532"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.397791 26727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:48.427040 26727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:48.429153 26727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:48.429271 26727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:48.429477 26727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:48.429788 26727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:48.429823 26727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.429850 26727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:48.429862 26727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.435392 26727 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:40785
I20240501 02:07:48.435443 26854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:40785 every 8 connection(s)
I20240501 02:07:48.435680 26727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:48.440347 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26727
I20240501 02:07:48.440475 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:48.440659 22577 external_mini_cluster.cc:1428] Setting key 54878e1ffdb3a173e88fb61e47d721da
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:48.444552 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.436401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:34141 (local address 127.22.12.126:41583)
0501 02:07:48.436485 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.436487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.436590 (+ 103us) server_negotiation.cc:407] Connection header received
0501 02:07:48.437093 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.437094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.437108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.437171 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:48.437979 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.438518 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.439108 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.439220 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.441558 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:48.441562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:48.441563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:48.441570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:48.443138 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.443569 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.443570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.443570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.443598 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.443857 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.443857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.443858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.443973 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:48.444036 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.444338 (+ 302us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.444437 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 02:07:48.444618 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.436326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41583 (local address 127.22.12.65:34141)
0501 02:07:48.436475 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.436504 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.436971 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.437263 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.437268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.437582 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:48.437875 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.437879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.438639 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.438640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.439030 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.439032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.439410 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.439955 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:48.439972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:48.441464 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:48.443231 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:48.443240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:48.443243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:48.443489 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:48.443690 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:48.443691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:48.443692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:48.443790 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:48.444054 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:48.444060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:48.444191 (+ 131us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.444296 (+ 105us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.444451 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20240501 02:07:48.445140 26855 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41583
I20240501 02:07:48.445255 26855 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:48.445467 26855 heartbeater.cc:511] Master 127.22.12.126:41583 requested a full tablet report, sending...
I20240501 02:07:48.446017 26665 ts_manager.cc:194] Registered new tserver with Master: 806abf8826a94f8b8d1b0cbd57dfcac5 (127.22.12.65:40785)
I20240501 02:07:48.446758 26665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:34141
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:07:48.467190 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41583
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:48.548146 26860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:48.548347 26860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:48.548370 26860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:48.549698 26860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:48.549739 26860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:48.549765 26860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:48.551450 26860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:48.551658 26860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:48.551816 26860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:48.556172 26866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.556615 26865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.556985 26868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:48.557440 26860 server_base.cc:1034] running on GCE node
I20240501 02:07:48.557610 26860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:48.557807 26860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:48.558933 26860 hybrid_clock.cc:648] HybridClock initialized: now 1714529268558916 us; error 26 us; skew 500 ppm
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:48.561262 26860 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:07:48.561568 26860 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:48.561609 26860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:48.561692 26860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:48.562686 26860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "b58272d0779c41538e136b381a5e917a"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "c0123fc028be84e51aa70bed78053f08"
server_key_iv: "ba5e8d96d8630d3f6c9d5db565b869f2"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.562949 26860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "b58272d0779c41538e136b381a5e917a"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "c0123fc028be84e51aa70bed78053f08"
server_key_iv: "ba5e8d96d8630d3f6c9d5db565b869f2"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.564005 26860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:48.564626 26874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.564787 26860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:48.564858 26860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "b58272d0779c41538e136b381a5e917a"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "c0123fc028be84e51aa70bed78053f08"
server_key_iv: "ba5e8d96d8630d3f6c9d5db565b869f2"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.564908 26860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:48.578390 26860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:48.580482 26860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:48.580587 26860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:48.580776 26860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:48.581075 26860 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:48.581108 26860 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.581128 26860 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:48.581146 26860 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.586951 26860 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:33017
I20240501 02:07:48.587003 26987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:33017 every 8 connection(s)
I20240501 02:07:48.587232 26860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:48.591939 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26860
I20240501 02:07:48.592054 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:48.592195 22577 external_mini_cluster.cc:1428] Setting key ea3815ea0294aecf308d21c7522f1522
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:48.595304 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.587911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:33507 (local address 127.22.12.126:41583)
0501 02:07:48.587977 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.587979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.588143 (+ 164us) server_negotiation.cc:407] Connection header received
0501 02:07:48.588598 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.588599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.588609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.588679 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:48.589454 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.589950 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.590494 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.590658 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.592848 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:48.592853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:48.592853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:48.592860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:48.594183 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.594547 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.594549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.594549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.594577 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.594733 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.594733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.594734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.594830 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:48.594885 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.595122 (+ 237us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.595213 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 02:07:48.595427 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.587889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41583 (local address 127.22.12.66:33507)
0501 02:07:48.588039 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.588064 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.588501 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.588748 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.588752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.589059 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:48.589339 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.589343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.590051 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.590052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.590428 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.590430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.590798 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.591331 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:48.591349 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:48.592757 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:48.594260 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:48.594266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:48.594269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:48.594465 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:48.594629 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:48.594629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:48.594630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:48.594685 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:48.594880 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:48.594885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:48.595004 (+ 119us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.595099 (+ 95us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.595241 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20240501 02:07:48.595890 26988 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41583
I20240501 02:07:48.595973 26988 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:48.596138 26988 heartbeater.cc:511] Master 127.22.12.126:41583 requested a full tablet report, sending...
I20240501 02:07:48.596499 26665 ts_manager.cc:194] Registered new tserver with Master: b58272d0779c41538e136b381a5e917a (127.22.12.66:33017)
I20240501 02:07:48.596849 26665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:33507
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:07:48.618798 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41583
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:07:48.698055 26993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:48.698276 26993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:48.698310 26993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:48.699784 26993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:48.699843 26993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:48.699882 26993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:48.701949 26993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:48.702212 26993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:48.702410 26993 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:48.707201 26999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.707590 26998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:48.708115 27001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:48.708618 26993 server_base.cc:1034] running on GCE node
I20240501 02:07:48.708834 26993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:48.709065 26993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:48.710227 26993 hybrid_clock.cc:648] HybridClock initialized: now 1714529268710209 us; error 33 us; skew 500 ppm
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:07:48.712826 26993 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:07:48.713155 26993 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:48.713200 26993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:48.713300 26993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:48.714337 26993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "e4f5bfbeff1e47739d94aecf7e7726fd"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "1a67147f39f700460841e829f2be8df0"
server_key_iv: "a7568247827d17a89d8d2d703f0ac8f2"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.714673 26993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "e4f5bfbeff1e47739d94aecf7e7726fd"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "1a67147f39f700460841e829f2be8df0"
server_key_iv: "a7568247827d17a89d8d2d703f0ac8f2"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.715874 26993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:48.716676 27007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.716861 26993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:07:48.716949 26993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "e4f5bfbeff1e47739d94aecf7e7726fd"
format_stamp: "Formatted at 2024-05-01 02:07:48 on dist-test-slave-9tkh"
server_key: "1a67147f39f700460841e829f2be8df0"
server_key_iv: "a7568247827d17a89d8d2d703f0ac8f2"
server_key_version: "encryptionkey@0"
I20240501 02:07:48.717010 26993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:48.727517 26993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:48.729916 26993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:48.730038 26993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:48.730309 26993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:48.730679 26993 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:48.730715 26993 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.730743 26993 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:48.730759 26993 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:48.736265 26993 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:35011
I20240501 02:07:48.736311 27120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:35011 every 8 connection(s)
I20240501 02:07:48.736544 26993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:07:48 dist-test-slave-9tkh krb5kdc[26621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:07:48.743314 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 26993
I20240501 02:07:48.743428 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:48.743607 22577 external_mini_cluster.cc:1428] Setting key 304d3e5513dd2a6c226bc203d894a7da
I20240501 02:07:48.744844 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.737260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:46533 (local address 127.22.12.126:41583)
0501 02:07:48.737317 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.737319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.737476 (+ 157us) server_negotiation.cc:407] Connection header received
0501 02:07:48.737920 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.737921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.737931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.737991 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:48.738793 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.739280 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.739835 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.739948 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.742086 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:48.742091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:48.742091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:48.742097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:48.743601 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.744014 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.744015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.744016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.744044 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.744247 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.744248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.744248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.744354 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:48.744410 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.744705 (+ 295us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.744778 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:07:48.744961 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.737186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41583 (local address 127.22.12.67:46533)
0501 02:07:48.737350 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.737377 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.737831 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.738057 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.738061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.738382 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:48.738716 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.738721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.739371 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.739372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.739762 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.739763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.740111 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.740659 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:07:48.740675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:48.741997 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:48.743698 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:48.743705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:48.743708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:48.743939 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:48.744104 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:48.744105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:07:48.744105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:07:48.744192 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:48.744410 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:48.744415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:48.744587 (+ 172us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.744683 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.744795 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20240501 02:07:48.745450 27121 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41583
I20240501 02:07:48.745548 27121 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:48.745754 27121 heartbeater.cc:511] Master 127.22.12.126:41583 requested a full tablet report, sending...
I20240501 02:07:48.746129 26665 ts_manager.cc:194] Registered new tserver with Master: e4f5bfbeff1e47739d94aecf7e7726fd (127.22.12.67:35011)
I20240501 02:07:48.746644 26665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:46533
I20240501 02:07:48.755365 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:07:48.761710 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.756609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.22.12.126:41583)
0501 02:07:48.756691 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.756694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.756735 (+ 41us) server_negotiation.cc:407] Connection header received
0501 02:07:48.756801 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.756802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.756812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.756856 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:48.757528 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.758003 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.758572 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.758738 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.759654 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:48.759658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:48.759658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:07:48.759665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:48.760803 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.761088 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.761089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.761089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.761111 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:48.761235 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:07:48.761236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:07:48.761236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:07:48.761348 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:48.761437 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.761532 (+ 95us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.761615 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 02:07:48.763458 26665 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:48.764941 26665 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:48.770630 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.767276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:33017 (local address 127.0.0.1:36418)
0501 02:07:48.767453 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.767475 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.767579 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.768025 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.768026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.768028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:48.768238 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.768240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.769569 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.769570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.770362 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.770364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.770419 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.770423 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.770451 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.770491 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":51,"threads_started":1}
I20240501 02:07:48.770870 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.767500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.22.12.65:40785)
0501 02:07:48.767698 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.767707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.767826 (+ 119us) server_negotiation.cc:407] Connection header received
0501 02:07:48.767869 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.767871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.767884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.767950 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:48.768269 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.769147 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.770098 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.770647 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.770706 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.770742 (+ 36us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.770789 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20240501 02:07:48.770753 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.767278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:35011 (local address 127.0.0.1:35562)
0501 02:07:48.767675 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.767684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.767721 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.768064 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.768064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.768067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:48.768463 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.768465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.769715 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.769716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.770470 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.770472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.770580 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.770585 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.770616 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.770654 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":90,"threads_started":1}
I20240501 02:07:48.770630 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.767275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:40785 (local address 127.0.0.1:35464)
0501 02:07:48.767771 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.767777 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.767816 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.767963 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.767966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.767967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:48.768199 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.768201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.769221 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.769222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.769978 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.769980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.770064 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.770107 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.770450 (+ 343us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.770491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":6,"thread_start_us":95,"threads_started":1}
I20240501 02:07:48.771412 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.767542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.22.12.67:35011)
0501 02:07:48.767834 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.767841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.767849 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:07:48.767880 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.767882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.767898 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.767984 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:48.768555 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.769644 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.770706 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.771215 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.771267 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.771301 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.771340 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
I20240501 02:07:48.771822 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.767685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.22.12.66:33017)
0501 02:07:48.767821 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.767829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.767841 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:07:48.767880 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.767882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.767898 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.768319 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:48.768392 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.769488 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.771121 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.771625 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.771671 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.771702 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.771743 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20240501 02:07:48.771862 26789 tablet_service.cc:1467] Processing CreateTablet for tablet c8f79bc00fa54895840ec62c1750d34d (DEFAULT_TABLE table=test-table [id=94dec0aa6e744ced8389685aa5d0b422]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:48.771998 27055 tablet_service.cc:1467] Processing CreateTablet for tablet c8f79bc00fa54895840ec62c1750d34d (DEFAULT_TABLE table=test-table [id=94dec0aa6e744ced8389685aa5d0b422]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:48.772167 26789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f79bc00fa54895840ec62c1750d34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:48.772270 27055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f79bc00fa54895840ec62c1750d34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:48.772331 26922 tablet_service.cc:1467] Processing CreateTablet for tablet c8f79bc00fa54895840ec62c1750d34d (DEFAULT_TABLE table=test-table [id=94dec0aa6e744ced8389685aa5d0b422]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:48.772569 26922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f79bc00fa54895840ec62c1750d34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:48.774861 27138 tablet_bootstrap.cc:492] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Bootstrap starting.
I20240501 02:07:48.775288 27140 tablet_bootstrap.cc:492] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Bootstrap starting.
I20240501 02:07:48.775664 27138 tablet_bootstrap.cc:654] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:48.775799 27140 tablet_bootstrap.cc:654] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:48.775966 27138 log.cc:824] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:48.776085 27140 log.cc:824] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:48.776378 27139 tablet_bootstrap.cc:492] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: Bootstrap starting.
I20240501 02:07:48.776717 27138 tablet_bootstrap.cc:492] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: No bootstrap required, opened a new log
I20240501 02:07:48.776748 27140 tablet_bootstrap.cc:492] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: No bootstrap required, opened a new log
I20240501 02:07:48.776783 27138 ts_tablet_manager.cc:1367] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:48.776800 27140 ts_tablet_manager.cc:1367] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:48.776921 27139 tablet_bootstrap.cc:654] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:48.777216 27139 log.cc:824] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:48.778009 27139 tablet_bootstrap.cc:492] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: No bootstrap required, opened a new log
I20240501 02:07:48.778077 27139 ts_tablet_manager.cc:1367] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:48.778649 27138 raft_consensus.cc:348] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.778777 27138 raft_consensus.cc:374] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:48.778800 27138 raft_consensus.cc:729] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f5bfbeff1e47739d94aecf7e7726fd, State: Initialized, Role: FOLLOWER
I20240501 02:07:48.778908 27138 consensus_queue.cc:260] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.778965 27140 raft_consensus.cc:348] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.779176 27140 raft_consensus.cc:374] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:48.779242 27140 raft_consensus.cc:729] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58272d0779c41538e136b381a5e917a, State: Initialized, Role: FOLLOWER
I20240501 02:07:48.779242 27138 ts_tablet_manager.cc:1398] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:07:48.779366 27121 heartbeater.cc:503] Master 127.22.12.126:41583 was elected leader, sending a full tablet report...
I20240501 02:07:48.779400 27140 consensus_queue.cc:260] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.779680 27140 ts_tablet_manager.cc:1398] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:07:48.779714 26988 heartbeater.cc:503] Master 127.22.12.126:41583 was elected leader, sending a full tablet report...
I20240501 02:07:48.780313 27139 raft_consensus.cc:348] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.780504 27139 raft_consensus.cc:374] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:48.780544 27139 raft_consensus.cc:729] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806abf8826a94f8b8d1b0cbd57dfcac5, State: Initialized, Role: FOLLOWER
I20240501 02:07:48.780651 27139 consensus_queue.cc:260] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.780894 27139 ts_tablet_manager.cc:1398] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:07:48.780970 26855 heartbeater.cc:503] Master 127.22.12.126:41583 was elected leader, sending a full tablet report...
W20240501 02:07:48.837776 26989 tablet.cc:2369] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:07:48.936411 26856 tablet.cc:2369] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:48.961400 27144 raft_consensus.cc:482] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:48.961540 27144 raft_consensus.cc:504] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.961875 27144 leader_election.cc:288] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58272d0779c41538e136b381a5e917a (127.22.12.66:33017), 806abf8826a94f8b8d1b0cbd57dfcac5 (127.22.12.65:40785)
I20240501 02:07:48.964773 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.962065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:33017 (local address 127.22.12.67:40633)
0501 02:07:48.962154 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.962164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.962210 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.962362 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.962363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.962364 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:48.962611 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.962613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.963552 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.963553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.964490 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.964492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.964569 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.964574 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.964604 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.964638 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20240501 02:07:48.964780 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.962074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:40785 (local address 127.22.12.67:47215)
0501 02:07:48.962254 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.962262 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:48.962310 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:48.962452 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:48.962453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:48.962454 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:07:48.962702 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.962703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.963567 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.963568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:48.964501 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:48.964502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.964570 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.964574 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:07:48.964604 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 02:07:48.964641 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":9,"thread_start_us":91,"threads_started":1}
I20240501 02:07:48.965162 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.962144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:47215 (local address 127.22.12.65:40785)
0501 02:07:48.962194 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.962197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.962342 (+ 145us) server_negotiation.cc:407] Connection header received
0501 02:07:48.962366 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.962367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.962394 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.962438 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:48.962773 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.963467 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.964575 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.965036 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.965045 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.965067 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.965097 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:07:48.965168 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.962143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:40633 (local address 127.22.12.66:33017)
0501 02:07:48.962197 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.962199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.962209 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:07:48.962289 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.962290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.962298 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.962349 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:07:48.962686 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.963454 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.964575 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.965044 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.965050 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.965067 (+ 17us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.965096 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:07:48.965482 26942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f79bc00fa54895840ec62c1750d34d" candidate_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58272d0779c41538e136b381a5e917a" is_pre_election: true
I20240501 02:07:48.965482 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f79bc00fa54895840ec62c1750d34d" candidate_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" is_pre_election: true
I20240501 02:07:48.965633 26942 raft_consensus.cc:2454] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f5bfbeff1e47739d94aecf7e7726fd in term 0.
I20240501 02:07:48.965633 26809 raft_consensus.cc:2454] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f5bfbeff1e47739d94aecf7e7726fd in term 0.
I20240501 02:07:48.965850 27011 leader_election.cc:302] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b58272d0779c41538e136b381a5e917a, e4f5bfbeff1e47739d94aecf7e7726fd; no voters:
I20240501 02:07:48.966022 27144 raft_consensus.cc:2781] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:48.966080 27144 raft_consensus.cc:482] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:48.966096 27144 raft_consensus.cc:3037] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:48.966746 27144 raft_consensus.cc:504] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.966948 27144 leader_election.cc:288] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [CANDIDATE]: Term 1 election: Requested vote from peers b58272d0779c41538e136b381a5e917a (127.22.12.66:33017), 806abf8826a94f8b8d1b0cbd57dfcac5 (127.22.12.65:40785)
I20240501 02:07:48.967041 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f79bc00fa54895840ec62c1750d34d" candidate_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5"
I20240501 02:07:48.967125 26809 raft_consensus.cc:3037] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:48.967129 26942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f79bc00fa54895840ec62c1750d34d" candidate_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58272d0779c41538e136b381a5e917a"
I20240501 02:07:48.967192 26942 raft_consensus.cc:3037] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:48.967973 26942 raft_consensus.cc:2454] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f5bfbeff1e47739d94aecf7e7726fd in term 1.
I20240501 02:07:48.967985 26809 raft_consensus.cc:2454] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f5bfbeff1e47739d94aecf7e7726fd in term 1.
I20240501 02:07:48.968202 27010 leader_election.cc:302] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806abf8826a94f8b8d1b0cbd57dfcac5, e4f5bfbeff1e47739d94aecf7e7726fd; no voters:
I20240501 02:07:48.968340 27144 raft_consensus.cc:2781] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:48.968508 27144 raft_consensus.cc:686] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 1 LEADER]: Becoming Leader. State: Replica: e4f5bfbeff1e47739d94aecf7e7726fd, State: Running, Role: LEADER
I20240501 02:07:48.968608 27144 consensus_queue.cc:237] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } }
I20240501 02:07:48.969470 26665 catalog_manager.cc:5484] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd reported cstate change: term changed from 0 to 1, leader changed from <none> to e4f5bfbeff1e47739d94aecf7e7726fd (127.22.12.67). New cstate: current_term: 1 leader_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f5bfbeff1e47739d94aecf7e7726fd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 35011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 } health_report { overall_health: UNKNOWN } } }
W20240501 02:07:48.987025 27122 tablet.cc:2369] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:48.998005 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:48.996001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.22.12.67:35011)
0501 02:07:48.996057 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:48.996059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:48.996066 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:07:48.996107 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:48.996108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:48.996125 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:48.996191 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:07:48.996388 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.996871 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:48.997513 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:48.997660 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:48.997698 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:48.997786 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:48.997856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:48.997894 (+ 38us) server_negotiation.cc:299] Negotiation successful
0501 02:07:48.997916 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:07:48.999663 26942 raft_consensus.cc:1261] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 1 FOLLOWER]: Refusing update from remote peer e4f5bfbeff1e47739d94aecf7e7726fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:48.999686 26809 raft_consensus.cc:1261] T c8f79bc00fa54895840ec62c1750d34d P 806abf8826a94f8b8d1b0cbd57dfcac5 [term 1 FOLLOWER]: Refusing update from remote peer e4f5bfbeff1e47739d94aecf7e7726fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:49.000005 27148 consensus_queue.cc:1035] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "806abf8826a94f8b8d1b0cbd57dfcac5" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:49.000125 27144 consensus_queue.cc:1035] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58272d0779c41538e136b381a5e917a" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 33017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:49.003268 27153 mvcc.cc:205] Tried to move back new op lower bound from 7022711885820243968 to 7022711885696122880. Current Snapshot: MvccSnapshot[applied={T|T < 7022711885820243968 or (T in {7022711885820243968})}]
I20240501 02:07:49.004698 26664 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45274:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:49.004782 26664 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45274:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:49.005538 26664 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1bd19ae9626b43f1ad15c4c1f5bd96c9: Sending DeleteTablet for 3 replicas of tablet c8f79bc00fa54895840ec62c1750d34d
I20240501 02:07:49.005823 27055 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f79bc00fa54895840ec62c1750d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:48 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:35562
I20240501 02:07:49.005910 26922 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f79bc00fa54895840ec62c1750d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:48 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:36418
I20240501 02:07:49.005914 26789 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f79bc00fa54895840ec62c1750d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:48 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:35464
I20240501 02:07:49.006035 27161 tablet_replica.cc:330] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: stopping tablet replica
I20240501 02:07:49.006103 27161 raft_consensus.cc:2229] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:49.006139 27162 tablet_replica.cc:330] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: stopping tablet replica
I20240501 02:07:49.006206 27162 raft_consensus.cc:2229] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:49.006234 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26727
I20240501 02:07:49.006317 27162 raft_consensus.cc:2258] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:49.006968 27161 raft_consensus.cc:2258] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:49.007388 27162 ts_tablet_manager.cc:1875] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:49.007388 27161 ts_tablet_manager.cc:1875] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:49.009145 27161 ts_tablet_manager.cc:1888] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:49.009145 27162 ts_tablet_manager.cc:1888] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:49.009207 27161 log.cc:1196] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/c8f79bc00fa54895840ec62c1750d34d
I20240501 02:07:49.009207 27162 log.cc:1196] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/c8f79bc00fa54895840ec62c1750d34d
I20240501 02:07:49.009464 27162 ts_tablet_manager.cc:1909] T c8f79bc00fa54895840ec62c1750d34d P b58272d0779c41538e136b381a5e917a: Deleting consensus metadata
I20240501 02:07:49.009470 27161 ts_tablet_manager.cc:1909] T c8f79bc00fa54895840ec62c1750d34d P e4f5bfbeff1e47739d94aecf7e7726fd: Deleting consensus metadata
I20240501 02:07:49.009837 26650 catalog_manager.cc:4830] TS e4f5bfbeff1e47739d94aecf7e7726fd (127.22.12.67:35011): tablet c8f79bc00fa54895840ec62c1750d34d (table test-table [id=94dec0aa6e744ced8389685aa5d0b422]) successfully deleted
I20240501 02:07:49.009847 26652 catalog_manager.cc:4830] TS b58272d0779c41538e136b381a5e917a (127.22.12.66:33017): tablet c8f79bc00fa54895840ec62c1750d34d (table test-table [id=94dec0aa6e744ced8389685aa5d0b422]) successfully deleted
W20240501 02:07:49.013090 26651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:40785 (error 108)
I20240501 02:07:49.013451 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26860
I20240501 02:07:49.013593 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.013475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:40785 (local address 127.0.0.1:35476)
0501 02:07:49.013526 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.013557 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240501 02:07:49.015377 26651 catalog_manager.cc:4573] TS 806abf8826a94f8b8d1b0cbd57dfcac5 (127.22.12.65:40785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8f79bc00fa54895840ec62c1750d34d: Network error: Client connection negotiation failed: client connection to 127.22.12.65:40785: connect: Connection refused (error 111)
I20240501 02:07:49.018358 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26993
I20240501 02:07:49.022691 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 26634
2024-05-01T02:07:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1940 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2024-05-01T02:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:49Z Disabled control of system clock
I20240501 02:07:49.048583 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:34855
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:34855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240501 02:07:49.126518 27168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:49.126765 27168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:49.126796 27168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:49.128165 27168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:49.128217 27168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:49.128233 27168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:49.128249 27168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:49.128263 27168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:07:49.129832 27168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:34855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:34855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:49.130087 27168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:49.130342 27168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:49.134939 27176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.135318 27173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.135774 27174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:49.136277 27168 server_base.cc:1034] running on GCE node
I20240501 02:07:49.136543 27168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:49.136817 27168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:49.137979 27168 hybrid_clock.cc:648] HybridClock initialized: now 1714529269137959 us; error 40 us; skew 500 ppm
I20240501 02:07:49.139379 27168 webserver.cc:460] Webserver started at http://127.22.12.126:35093/ using document root <none> and password file <none>
I20240501 02:07:49.139559 27168 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:49.139629 27168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:49.139734 27168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:49.140738 27168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "180c4becf109423589d652bd622cb431"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "01b3a0c0d98b303b1db0c6fe5cea38a2"
server_key_iv: "2b22ea99534cc59479b6aa49faa9b096"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.141023 27168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "180c4becf109423589d652bd622cb431"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "01b3a0c0d98b303b1db0c6fe5cea38a2"
server_key_iv: "2b22ea99534cc59479b6aa49faa9b096"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.142213 27168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:07:49.143009 27182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.143201 27168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:07:49.143289 27168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "180c4becf109423589d652bd622cb431"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "01b3a0c0d98b303b1db0c6fe5cea38a2"
server_key_iv: "2b22ea99534cc59479b6aa49faa9b096"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.143354 27168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:49.147071 27168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:49.147339 27168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:49.147442 27168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:49.151075 27168 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:34855
I20240501 02:07:49.151114 27234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:34855 every 8 connection(s)
I20240501 02:07:49.151404 27168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:49.151973 27235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:49.152273 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27168
I20240501 02:07:49.152365 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:49.152544 22577 external_mini_cluster.cc:1428] Setting key 2b998aeaf3a11a11379aecd476c01288
I20240501 02:07:49.154982 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.153078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.22.12.126:34855)
0501 02:07:49.153241 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.153248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.153266 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:07:49.153727 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.153733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.153770 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.153892 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.154612 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.154615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.154616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.154619 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.154647 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.154660 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.154712 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.154794 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.154836 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20240501 02:07:49.155480 27235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431: Bootstrap starting.
I20240501 02:07:49.155987 27235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:49.156231 27235 log.cc:824] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:49.156886 27235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431: No bootstrap required, opened a new log
I20240501 02:07:49.158156 27235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180c4becf109423589d652bd622cb431" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34855 } }
I20240501 02:07:49.158290 27235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:49.158308 27235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180c4becf109423589d652bd622cb431, State: Initialized, Role: FOLLOWER
I20240501 02:07:49.158385 27235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180c4becf109423589d652bd622cb431" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34855 } }
I20240501 02:07:49.158445 27235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:49.158465 27235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:49.158497 27235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:49.159035 27235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180c4becf109423589d652bd622cb431" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34855 } }
I20240501 02:07:49.159122 27235 leader_election.cc:302] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 180c4becf109423589d652bd622cb431; no voters:
I20240501 02:07:49.159267 27235 leader_election.cc:288] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:49.159332 27240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:49.159473 27240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [term 1 LEADER]: Becoming Leader. State: Replica: 180c4becf109423589d652bd622cb431, State: Running, Role: LEADER
I20240501 02:07:49.159495 27235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:49.159597 27240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180c4becf109423589d652bd622cb431" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34855 } }
I20240501 02:07:49.159952 27241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "180c4becf109423589d652bd622cb431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180c4becf109423589d652bd622cb431" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34855 } } }
I20240501 02:07:49.159982 27242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 180c4becf109423589d652bd622cb431. Latest consensus state: current_term: 1 leader_uuid: "180c4becf109423589d652bd622cb431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180c4becf109423589d652bd622cb431" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34855 } } }
I20240501 02:07:49.160037 27241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:49.160059 27242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:49.160277 27245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:49.160769 27245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:49.162145 27245 catalog_manager.cc:1347] Generated new cluster ID: 0e97a17b24e242ae88b8625a2288219f
I20240501 02:07:49.162200 27245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:49.188437 27245 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:49.188724 27245 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:49.196094 27245 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431: Generated new TSK 0
I20240501 02:07:49.196367 27245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:07:49.204591 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34855
--builtin_ntp_servers=127.22.12.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240501 02:07:49.296716 27259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:49.296913 27259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:49.296936 27259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:49.298292 27259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:49.298338 27259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:49.298362 27259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:07:49.299942 27259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:49.300155 27259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:49.300391 27259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:49.305368 27265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.305677 27267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.306165 27264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:49.306603 27259 server_base.cc:1034] running on GCE node
I20240501 02:07:49.306808 27259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:49.307022 27259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:49.308193 27259 hybrid_clock.cc:648] HybridClock initialized: now 1714529269308179 us; error 31 us; skew 500 ppm
I20240501 02:07:49.309613 27259 webserver.cc:460] Webserver started at http://127.22.12.65:36943/ using document root <none> and password file <none>
I20240501 02:07:49.309788 27259 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:49.309829 27259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:49.309944 27259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:49.311020 27259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "62c870e74dc1415d8925bb5954ee7663"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "7bf77a477744780071edf75b81e8cf18"
server_key_iv: "e331a60bd1aaaa8d7037a1d84adca20f"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.311332 27259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "62c870e74dc1415d8925bb5954ee7663"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "7bf77a477744780071edf75b81e8cf18"
server_key_iv: "e331a60bd1aaaa8d7037a1d84adca20f"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.312450 27259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:07:49.313119 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.313334 27259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:49.313438 27259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "62c870e74dc1415d8925bb5954ee7663"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "7bf77a477744780071edf75b81e8cf18"
server_key_iv: "e331a60bd1aaaa8d7037a1d84adca20f"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.313510 27259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:49.318688 27259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:49.318964 27259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:49.319062 27259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:49.319269 27259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:49.319581 27259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:49.319622 27259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.319658 27259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:49.319681 27259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.326296 27259 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:37523
I20240501 02:07:49.326332 27386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:37523 every 8 connection(s)
I20240501 02:07:49.326756 27259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:07:49.329706 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.327543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:45741 (local address 127.22.12.126:34855)
0501 02:07:49.327610 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.327612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.327853 (+ 241us) server_negotiation.cc:407] Connection header received
0501 02:07:49.328474 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.328476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.328483 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.328552 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.329450 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.329452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.329452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.329457 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.329473 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.329476 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.329524 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.329596 (+ 72us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.329611 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:07:49.329872 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.327508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:34855 (local address 127.22.12.65:45741)
0501 02:07:49.327734 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.327764 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.328334 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.328690 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.328695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.328705 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.329278 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.329278 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.329307 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.329350 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.329527 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.329531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.329534 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.329631 (+ 97us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.329684 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20240501 02:07:49.330019 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27259
I20240501 02:07:49.330129 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:07:49.330318 22577 external_mini_cluster.cc:1428] Setting key 51dd506d5d6e522a5bc7dd71abc2e532
I20240501 02:07:49.330672 27387 heartbeater.cc:347] Connected to a master server at 127.22.12.126:34855
I20240501 02:07:49.330770 27387 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:49.330900 27387 heartbeater.cc:511] Master 127.22.12.126:34855 requested a full tablet report, sending...
I20240501 02:07:49.331380 27199 ts_manager.cc:194] Registered new tserver with Master: 62c870e74dc1415d8925bb5954ee7663 (127.22.12.65:37523)
I20240501 02:07:49.331465 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34855
--builtin_ntp_servers=127.22.12.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240501 02:07:49.411243 27390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:49.411435 27390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:49.411466 27390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:49.412778 27390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:49.412827 27390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:49.412858 27390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:07:49.414317 27390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:49.414584 27390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:49.414834 27390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:49.419781 27396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.420150 27398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.420569 27395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:49.421079 27390 server_base.cc:1034] running on GCE node
I20240501 02:07:49.421267 27390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:49.421484 27390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:49.422612 27390 hybrid_clock.cc:648] HybridClock initialized: now 1714529269422593 us; error 35 us; skew 500 ppm
I20240501 02:07:49.424033 27390 webserver.cc:460] Webserver started at http://127.22.12.66:37109/ using document root <none> and password file <none>
I20240501 02:07:49.424230 27390 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:49.424273 27390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:49.424377 27390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:49.425354 27390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "6eb69ae290d84d0692abf1ed2f4e45a9"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "ee434a244a4dfb49447da67603fb1c0b"
server_key_iv: "a249a42287a33c176c81fb635586b8c9"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.425644 27390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "6eb69ae290d84d0692abf1ed2f4e45a9"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "ee434a244a4dfb49447da67603fb1c0b"
server_key_iv: "a249a42287a33c176c81fb635586b8c9"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.426820 27390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:07:49.427501 27404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.427654 27390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:49.427728 27390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "6eb69ae290d84d0692abf1ed2f4e45a9"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "ee434a244a4dfb49447da67603fb1c0b"
server_key_iv: "a249a42287a33c176c81fb635586b8c9"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.427793 27390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:49.431314 27390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:49.431541 27390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:49.431638 27390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:49.431820 27390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:49.432090 27390 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:49.432121 27390 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.432147 27390 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:49.432178 27390 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.437428 27390 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:37663
I20240501 02:07:49.437530 27517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:37663 every 8 connection(s)
I20240501 02:07:49.437779 27390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:07:49.440343 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.438492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:37667 (local address 127.22.12.126:34855)
0501 02:07:49.438615 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.438618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.438779 (+ 161us) server_negotiation.cc:407] Connection header received
0501 02:07:49.439239 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.439241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.439247 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.439336 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.440059 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.440061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.440061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.440065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.440080 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.440084 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.440136 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.440269 (+ 133us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.440283 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20240501 02:07:49.440490 27520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.438479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:34855 (local address 127.22.12.66:37667)
0501 02:07:49.438675 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.438704 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.439133 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.439384 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.439387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.439396 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.439901 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.439902 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.439922 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.439944 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.440181 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.440186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.440188 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.440286 (+ 98us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.440329 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
I20240501 02:07:49.441135 27518 heartbeater.cc:347] Connected to a master server at 127.22.12.126:34855
I20240501 02:07:49.441224 27518 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:49.441340 27518 heartbeater.cc:511] Master 127.22.12.126:34855 requested a full tablet report, sending...
I20240501 02:07:49.441735 27199 ts_manager.cc:194] Registered new tserver with Master: 6eb69ae290d84d0692abf1ed2f4e45a9 (127.22.12.66:37663)
I20240501 02:07:49.445878 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27390
I20240501 02:07:49.445997 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:07:49.446167 22577 external_mini_cluster.cc:1428] Setting key c469600e6067d1636e578c5c29d13621
I20240501 02:07:49.447075 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34855
--builtin_ntp_servers=127.22.12.84:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240501 02:07:49.525856 27521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:49.526046 27521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:49.526067 27521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:49.527410 27521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:49.527454 27521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:49.527478 27521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:07:49.528913 27521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:49.529115 27521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:49.529332 27521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:49.534085 27526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.534482 27527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:49.534833 27529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:49.535439 27521 server_base.cc:1034] running on GCE node
I20240501 02:07:49.535677 27521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:49.535904 27521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:49.537068 27521 hybrid_clock.cc:648] HybridClock initialized: now 1714529269537048 us; error 34 us; skew 500 ppm
I20240501 02:07:49.538522 27521 webserver.cc:460] Webserver started at http://127.22.12.67:46737/ using document root <none> and password file <none>
I20240501 02:07:49.538743 27521 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:49.538781 27521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:49.538869 27521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:49.539844 27521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "b54ec045ded94bb2b4d3ee99f72d37b5"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "3a98e7cab144f7b122d41f008ee72502"
server_key_iv: "cf332d4bbf0e56df349b299da169d0c4"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.540212 27521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "b54ec045ded94bb2b4d3ee99f72d37b5"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "3a98e7cab144f7b122d41f008ee72502"
server_key_iv: "cf332d4bbf0e56df349b299da169d0c4"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.541530 27521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 02:07:49.542142 27535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.542299 27521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:07:49.542376 27521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "b54ec045ded94bb2b4d3ee99f72d37b5"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "3a98e7cab144f7b122d41f008ee72502"
server_key_iv: "cf332d4bbf0e56df349b299da169d0c4"
server_key_version: "encryptionkey@0"
I20240501 02:07:49.542435 27521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:49.546070 27521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:49.546286 27521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:49.546386 27521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:49.546569 27521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:07:49.546857 27521 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:07:49.546887 27521 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.546913 27521 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:07:49.546931 27521 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:49.551965 27521 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:44625
I20240501 02:07:49.552002 27648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:44625 every 8 connection(s)
I20240501 02:07:49.552294 27521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:07:49.554729 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.552951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:47347 (local address 127.22.12.126:34855)
0501 02:07:49.553005 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.553007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.553201 (+ 194us) server_negotiation.cc:407] Connection header received
0501 02:07:49.553681 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.553682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.553688 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.553742 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.554445 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.554446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.554447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.554451 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.554467 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.554470 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.554517 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.554632 (+ 115us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.554646 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 02:07:49.554867 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.552935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:34855 (local address 127.22.12.67:47347)
0501 02:07:49.553101 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.553124 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.553578 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.553799 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.553803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.553811 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.554319 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.554319 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.554339 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.554362 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.554522 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.554561 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.554564 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.554657 (+ 93us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.554708 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20240501 02:07:49.555555 27649 heartbeater.cc:347] Connected to a master server at 127.22.12.126:34855
I20240501 02:07:49.555652 27649 heartbeater.cc:464] Registering TS with master...
I20240501 02:07:49.555754 27649 heartbeater.cc:511] Master 127.22.12.126:34855 requested a full tablet report, sending...
I20240501 02:07:49.556089 27199 ts_manager.cc:194] Registered new tserver with Master: b54ec045ded94bb2b4d3ee99f72d37b5 (127.22.12.67:44625)
I20240501 02:07:49.561256 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27521
I20240501 02:07:49.561359 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:07:49.561501 22577 external_mini_cluster.cc:1428] Setting key 10b2cde09b6edd9b08fe352aa4cd0f28
I20240501 02:07:49.562423 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:07:49.566213 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.563779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.22.12.126:34855)
0501 02:07:49.563819 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.563821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.563944 (+ 123us) server_negotiation.cc:407] Connection header received
0501 02:07:49.564214 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.564221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.564236 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.564368 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.565992 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.565993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.565993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.565997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.566012 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.566016 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.566065 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.566120 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.566135 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:07:49.568032 27199 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:07:49.569516 27199 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:07:49.573276 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.571539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:37663 (local address 127.0.0.1:54976)
0501 02:07:49.571714 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.571744 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.571804 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.572489 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.572490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.572493 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.572856 (+ 363us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.572857 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.572890 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.572914 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.573111 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.573116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.573119 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.573188 (+ 69us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.573202 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20240501 02:07:49.573325 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.572020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.22.12.66:37663)
0501 02:07:49.572167 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.572174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.572183 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:07:49.572206 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.572208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.572215 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.572475 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.572990 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.572994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.572995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.572999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.573040 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.573047 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.573114 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.573200 (+ 86us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.573213 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20240501 02:07:49.573462 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.571539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:44625 (local address 127.0.0.1:51532)
0501 02:07:49.571890 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.571898 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.571947 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.572106 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.572111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.572125 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.572883 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.572884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.572892 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.572913 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.573361 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.573361 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.573362 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.573400 (+ 38us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.573411 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20240501 02:07:49.573685 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.571664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.22.12.67:44625)
0501 02:07:49.571844 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.571850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.571953 (+ 103us) server_negotiation.cc:407] Connection header received
0501 02:07:49.572004 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.572006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.572016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.572082 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.572992 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.572999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.572999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.573003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.573046 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.573052 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.573124 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.573557 (+ 433us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.573572 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20240501 02:07:49.573738 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.571857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.22.12.65:37523)
0501 02:07:49.572029 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.572037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.572307 (+ 270us) server_negotiation.cc:407] Connection header received
0501 02:07:49.572403 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.572405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.572414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.572461 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.573435 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.573440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.573440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.573445 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.573484 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.573490 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.573531 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.573637 (+ 106us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.573651 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20240501 02:07:49.573879 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.571542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37523 (local address 127.0.0.1:50000)
0501 02:07:49.572240 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.572250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.572329 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.572893 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.572894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.572896 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.573333 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.573333 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.573339 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.573348 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.573573 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.573574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.573574 (+ 0us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.573805 (+ 231us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.573819 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":579,"thread_start_us":60,"threads_started":1}
I20240501 02:07:49.574204 27583 tablet_service.cc:1467] Processing CreateTablet for tablet ade708f560b44a55b53c60c5f9cb4cb9 (DEFAULT_TABLE table=test-table [id=97f1659837a7446ba217872f609cabfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:49.574208 27452 tablet_service.cc:1467] Processing CreateTablet for tablet ade708f560b44a55b53c60c5f9cb4cb9 (DEFAULT_TABLE table=test-table [id=97f1659837a7446ba217872f609cabfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:49.574487 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade708f560b44a55b53c60c5f9cb4cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:49.574491 27452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade708f560b44a55b53c60c5f9cb4cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:49.574431 27321 tablet_service.cc:1467] Processing CreateTablet for tablet ade708f560b44a55b53c60c5f9cb4cb9 (DEFAULT_TABLE table=test-table [id=97f1659837a7446ba217872f609cabfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:07:49.574705 27321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade708f560b44a55b53c60c5f9cb4cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:49.576792 27663 tablet_bootstrap.cc:492] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Bootstrap starting.
I20240501 02:07:49.577337 27663 tablet_bootstrap.cc:654] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:49.577423 27664 tablet_bootstrap.cc:492] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Bootstrap starting.
I20240501 02:07:49.577682 27663 log.cc:824] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:49.577840 27665 tablet_bootstrap.cc:492] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Bootstrap starting.
I20240501 02:07:49.578039 27664 tablet_bootstrap.cc:654] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:49.578310 27664 log.cc:824] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:49.578387 27665 tablet_bootstrap.cc:654] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:49.578428 27663 tablet_bootstrap.cc:492] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: No bootstrap required, opened a new log
I20240501 02:07:49.578495 27663 ts_tablet_manager.cc:1367] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:49.578814 27665 log.cc:824] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:49.579155 27664 tablet_bootstrap.cc:492] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: No bootstrap required, opened a new log
I20240501 02:07:49.579214 27664 ts_tablet_manager.cc:1367] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:49.579571 27665 tablet_bootstrap.cc:492] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: No bootstrap required, opened a new log
I20240501 02:07:49.579622 27665 ts_tablet_manager.cc:1367] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:07:49.580412 27663 raft_consensus.cc:348] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.580514 27663 raft_consensus.cc:374] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:49.580528 27663 raft_consensus.cc:729] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54ec045ded94bb2b4d3ee99f72d37b5, State: Initialized, Role: FOLLOWER
I20240501 02:07:49.580601 27663 consensus_queue.cc:260] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.580796 27663 ts_tablet_manager.cc:1398] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:07:49.580837 27649 heartbeater.cc:503] Master 127.22.12.126:34855 was elected leader, sending a full tablet report...
I20240501 02:07:49.581391 27664 raft_consensus.cc:348] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.581524 27664 raft_consensus.cc:374] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:49.581545 27664 raft_consensus.cc:729] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb69ae290d84d0692abf1ed2f4e45a9, State: Initialized, Role: FOLLOWER
I20240501 02:07:49.581641 27664 consensus_queue.cc:260] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.581722 27665 raft_consensus.cc:348] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.581843 27665 raft_consensus.cc:374] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:49.581866 27665 raft_consensus.cc:729] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c870e74dc1415d8925bb5954ee7663, State: Initialized, Role: FOLLOWER
I20240501 02:07:49.581877 27664 ts_tablet_manager.cc:1398] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:07:49.581979 27518 heartbeater.cc:503] Master 127.22.12.126:34855 was elected leader, sending a full tablet report...
I20240501 02:07:49.581974 27665 consensus_queue.cc:260] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.582191 27665 ts_tablet_manager.cc:1398] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:07:49.582316 27387 heartbeater.cc:503] Master 127.22.12.126:34855 was elected leader, sending a full tablet report...
W20240501 02:07:49.688328 27519 tablet.cc:2369] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:07:49.802789 27650 tablet.cc:2369] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:49.821942 27670 raft_consensus.cc:482] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:07:49.822070 27670 raft_consensus.cc:504] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.822369 27670 leader_election.cc:288] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b54ec045ded94bb2b4d3ee99f72d37b5 (127.22.12.67:44625), 62c870e74dc1415d8925bb5954ee7663 (127.22.12.65:37523)
I20240501 02:07:49.823638 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.822653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:43221 (local address 127.22.12.67:44625)
0501 02:07:49.822716 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.822719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.822730 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:07:49.822763 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.822764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.822771 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.822832 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.823437 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.823438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.823439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.823442 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.823455 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.823459 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.823501 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.823568 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.823582 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 02:07:49.823629 27520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.822597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:44625 (local address 127.22.12.66:43221)
0501 02:07:49.822662 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.822677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.822718 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.822854 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.822854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.822857 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.823343 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.823343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.823350 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.823360 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.823504 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.823504 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.823505 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.823548 (+ 43us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.823561 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 02:07:49.823835 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.822738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:34991 (local address 127.22.12.65:37523)
0501 02:07:49.822779 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.822781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.822863 (+ 82us) server_negotiation.cc:407] Connection header received
0501 02:07:49.822915 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.822916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.822922 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.822978 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.823593 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.823594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.823594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.823599 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.823613 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.823616 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.823665 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.823746 (+ 81us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.823764 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:07:49.824028 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.822628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37523 (local address 127.22.12.66:34991)
0501 02:07:49.822794 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.822803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:49.822856 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:49.823000 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:49.823001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:49.823002 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:07:49.823502 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:07:49.823502 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:07:49.823508 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:07:49.823518 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:07:49.823670 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:07:49.823670 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:07:49.823671 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:07:49.823935 (+ 264us) client_negotiation.cc:240] Negotiation successful
0501 02:07:49.823949 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20240501 02:07:49.823983 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade708f560b44a55b53c60c5f9cb4cb9" candidate_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" is_pre_election: true
I20240501 02:07:49.824157 27603 raft_consensus.cc:2454] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eb69ae290d84d0692abf1ed2f4e45a9 in term 0.
I20240501 02:07:49.824335 27341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade708f560b44a55b53c60c5f9cb4cb9" candidate_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c870e74dc1415d8925bb5954ee7663" is_pre_election: true
I20240501 02:07:49.824373 27408 leader_election.cc:302] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eb69ae290d84d0692abf1ed2f4e45a9, b54ec045ded94bb2b4d3ee99f72d37b5; no voters:
I20240501 02:07:49.824479 27341 raft_consensus.cc:2454] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eb69ae290d84d0692abf1ed2f4e45a9 in term 0.
I20240501 02:07:49.824479 27670 raft_consensus.cc:2781] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:07:49.824556 27670 raft_consensus.cc:482] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:07:49.824589 27670 raft_consensus.cc:3037] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:49.825273 27670 raft_consensus.cc:504] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
I20240501 02:07:49.825395 27670 leader_election.cc:288] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [CANDIDATE]: Term 1 election: Requested vote from peers b54ec045ded94bb2b4d3ee99f72d37b5 (127.22.12.67:44625), 62c870e74dc1415d8925bb5954ee7663 (127.22.12.65:37523)
I20240501 02:07:49.825572 27341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade708f560b44a55b53c60c5f9cb4cb9" candidate_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c870e74dc1415d8925bb5954ee7663"
I20240501 02:07:49.825642 27341 raft_consensus.cc:3037] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:49.825649 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade708f560b44a55b53c60c5f9cb4cb9" candidate_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5"
I20240501 02:07:49.825727 27603 raft_consensus.cc:3037] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:49.826364 27341 raft_consensus.cc:2454] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eb69ae290d84d0692abf1ed2f4e45a9 in term 1.
I20240501 02:07:49.826553 27603 raft_consensus.cc:2454] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eb69ae290d84d0692abf1ed2f4e45a9 in term 1.
I20240501 02:07:49.826572 27406 leader_election.cc:302] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c870e74dc1415d8925bb5954ee7663, 6eb69ae290d84d0692abf1ed2f4e45a9; no voters:
I20240501 02:07:49.826756 27670 raft_consensus.cc:2781] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:49.826925 27670 raft_consensus.cc:686] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6eb69ae290d84d0692abf1ed2f4e45a9, State: Running, Role: LEADER
I20240501 02:07:49.827054 27670 consensus_queue.cc:237] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } }
W20240501 02:07:49.827397 27388 tablet.cc:2369] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:07:49.827813 27199 catalog_manager.cc:5484] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eb69ae290d84d0692abf1ed2f4e45a9 (127.22.12.66). New cstate: current_term: 1 leader_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6eb69ae290d84d0692abf1ed2f4e45a9" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 37663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 } health_report { overall_health: UNKNOWN } } }
I20240501 02:07:49.860707 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.859750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.22.12.66:37663)
0501 02:07:49.859813 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:49.859815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:49.859822 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:07:49.859868 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:49.859869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:49.859880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:49.859924 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:49.860489 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:49.860490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:49.860491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:49.860495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:49.860510 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:49.860514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:49.860562 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:49.860619 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 02:07:49.860633 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:07:49.862427 27603 raft_consensus.cc:1261] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 1 FOLLOWER]: Refusing update from remote peer 6eb69ae290d84d0692abf1ed2f4e45a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:49.862427 27341 raft_consensus.cc:1261] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 1 FOLLOWER]: Refusing update from remote peer 6eb69ae290d84d0692abf1ed2f4e45a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:07:49.862797 27670 consensus_queue.cc:1035] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c870e74dc1415d8925bb5954ee7663" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:49.862927 27673 consensus_queue.cc:1035] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54ec045ded94bb2b4d3ee99f72d37b5" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:07:49.865234 27676 mvcc.cc:205] Tried to move back new op lower bound from 7022711889354055680 to 7022711889212112896. Current Snapshot: MvccSnapshot[applied={T|T < 7022711889354055680}]
I20240501 02:07:49.865543 27677 mvcc.cc:205] Tried to move back new op lower bound from 7022711889354055680 to 7022711889212112896. Current Snapshot: MvccSnapshot[applied={T|T < 7022711889354055680 or (T in {7022711889354055680})}]
I20240501 02:07:49.866103 27678 mvcc.cc:205] Tried to move back new op lower bound from 7022711889354055680 to 7022711889212112896. Current Snapshot: MvccSnapshot[applied={T|T < 7022711889354055680 or (T in {7022711889354055680})}]
I20240501 02:07:49.868290 27198 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39548:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:49.868384 27198 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39548:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:07:49.869233 27198 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 180c4becf109423589d652bd622cb431: Sending DeleteTablet for 3 replicas of tablet ade708f560b44a55b53c60c5f9cb4cb9
I20240501 02:07:49.869565 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet ade708f560b44a55b53c60c5f9cb4cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:49 UTC) from {username='slave'} at 127.0.0.1:51532
I20240501 02:07:49.869597 27321 tablet_service.cc:1514] Processing DeleteTablet for tablet ade708f560b44a55b53c60c5f9cb4cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:49 UTC) from {username='slave'} at 127.0.0.1:50000
I20240501 02:07:49.869774 27452 tablet_service.cc:1514] Processing DeleteTablet for tablet ade708f560b44a55b53c60c5f9cb4cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:07:49 UTC) from {username='slave'} at 127.0.0.1:54976
I20240501 02:07:49.869828 27686 tablet_replica.cc:330] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: stopping tablet replica
I20240501 02:07:49.869910 27686 raft_consensus.cc:2229] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:49.869985 27688 tablet_replica.cc:330] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: stopping tablet replica
I20240501 02:07:49.870007 27686 raft_consensus.cc:2258] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:49.870090 27688 raft_consensus.cc:2229] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:07:49.870221 27688 raft_consensus.cc:2258] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:07:49.870332 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27259
I20240501 02:07:49.870492 27686 ts_tablet_manager.cc:1875] T ade708f560b44a55b53c60c5f9cb4cb9 P 62c870e74dc1415d8925bb5954ee7663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:49.873054 27688 ts_tablet_manager.cc:1875] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:49.874989 27688 ts_tablet_manager.cc:1888] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:49.875082 27688 log.cc:1196] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/ade708f560b44a55b53c60c5f9cb4cb9
I20240501 02:07:49.875335 27688 ts_tablet_manager.cc:1909] T ade708f560b44a55b53c60c5f9cb4cb9 P 6eb69ae290d84d0692abf1ed2f4e45a9: Deleting consensus metadata
I20240501 02:07:49.875785 27185 catalog_manager.cc:4830] TS 6eb69ae290d84d0692abf1ed2f4e45a9 (127.22.12.66:37663): tablet ade708f560b44a55b53c60c5f9cb4cb9 (table test-table [id=97f1659837a7446ba217872f609cabfe]) successfully deleted
W20240501 02:07:49.876943 27184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:37523 (error 108)
I20240501 02:07:49.877451 27687 tablet_replica.cc:330] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: stopping tablet replica
I20240501 02:07:49.877471 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:49.877329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37523 (local address 127.0.0.1:50016)
0501 02:07:49.877386 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.877431 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240501 02:07:49.877568 27687 raft_consensus.cc:2229] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:07:49.877694 27687 raft_consensus.cc:2258] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 02:07:49.877682 27184 catalog_manager.cc:4573] TS 62c870e74dc1415d8925bb5954ee7663 (127.22.12.65:37523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ade708f560b44a55b53c60c5f9cb4cb9: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37523: connect: Connection refused (error 111)
I20240501 02:07:49.877882 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27390
I20240501 02:07:49.878376 27687 ts_tablet_manager.cc:1875] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:07:49.880410 27687 ts_tablet_manager.cc:1888] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:07:49.880494 27687 log.cc:1196] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/ade708f560b44a55b53c60c5f9cb4cb9
I20240501 02:07:49.880759 27687 ts_tablet_manager.cc:1909] T ade708f560b44a55b53c60c5f9cb4cb9 P b54ec045ded94bb2b4d3ee99f72d37b5: Deleting consensus metadata
I20240501 02:07:49.881224 27186 catalog_manager.cc:4830] TS b54ec045ded94bb2b4d3ee99f72d37b5 (127.22.12.67:44625): tablet ade708f560b44a55b53c60c5f9cb4cb9 (table test-table [id=97f1659837a7446ba217872f609cabfe]) successfully deleted
I20240501 02:07:49.883244 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27521
I20240501 02:07:49.887890 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27168
2024-05-01T02:07:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (864 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20240501 02:07:49.906606 22577 mini_oidc.cc:152] Starting JWKS server
I20240501 02:07:49.907649 22577 webserver.cc:460] Webserver started at https://127.0.0.1:42439/ using document root <none> and password file <none>
I20240501 02:07:49.907728 22577 mini_oidc.cc:189] Starting OIDC Discovery server
I20240501 02:07:49.907850 22577 webserver.cc:460] Webserver started at http://127.0.0.1:38611/ using document root <none> and password file <none>
2024-05-01T02:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:07:49Z Disabled control of system clock
I20240501 02:07:49.914943 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39575
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:39575
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/testchainca.cert with env {}
W20240501 02:07:49.993853 27695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:07:49.994041 27695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:07:49.994071 27695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:07:49.995400 27695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:07:49.995441 27695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:07:49.995452 27695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:07:49.995462 27695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:07:49.995472 27695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 02:07:49.995482 27695 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 02:07:49.995496 27695 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42439/jwks.json
I20240501 02:07:49.997128 27695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:39575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:07:49.997319 27695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:07:49.997457 27695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:07:50.001711 27703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:50.002112 27701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:07:50.002502 27700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:07:50.002948 27695 server_base.cc:1034] running on GCE node
I20240501 02:07:50.003196 27695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:07:50.003448 27695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:07:50.004591 27695 hybrid_clock.cc:648] HybridClock initialized: now 1714529270004556 us; error 51 us; skew 500 ppm
I20240501 02:07:50.005820 27695 webserver.cc:460] Webserver started at http://127.22.12.126:44587/ using document root <none> and password file <none>
I20240501 02:07:50.006022 27695 fs_manager.cc:359] Metadata directory not provided
I20240501 02:07:50.006064 27695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:07:50.006201 27695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:07:50.007233 27695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "cb033f1625cf412897f6cad52cb95390"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "b31bc5ff81a90957b7215bd27b614c33"
server_key_iv: "393a3e3f33e79b3e6409ce8ba474f62d"
server_key_version: "encryptionkey@0"
I20240501 02:07:50.007548 27695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "cb033f1625cf412897f6cad52cb95390"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "b31bc5ff81a90957b7215bd27b614c33"
server_key_iv: "393a3e3f33e79b3e6409ce8ba474f62d"
server_key_version: "encryptionkey@0"
I20240501 02:07:50.008683 27695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:07:50.009384 27709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:07:50.009548 27695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:07:50.009624 27695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "cb033f1625cf412897f6cad52cb95390"
format_stamp: "Formatted at 2024-05-01 02:07:49 on dist-test-slave-9tkh"
server_key: "b31bc5ff81a90957b7215bd27b614c33"
server_key_iv: "393a3e3f33e79b3e6409ce8ba474f62d"
server_key_version: "encryptionkey@0"
I20240501 02:07:50.009685 27695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:07:50.022445 27695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:07:50.022819 27695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:07:50.022933 27695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:07:50.026580 27695 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:39575
I20240501 02:07:50.026648 27761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:39575 every 8 connection(s)
I20240501 02:07:50.026928 27695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:07:50.027516 27762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:07:50.028918 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27695
I20240501 02:07:50.029027 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:07:50.029189 22577 external_mini_cluster.cc:1428] Setting key 9931efd5ab83237d9d0b71f8514b6619
I20240501 02:07:50.030417 27762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390: Bootstrap starting.
I20240501 02:07:50.030963 27762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390: Neither blocks nor log segments found. Creating new log.
I20240501 02:07:50.031203 27762 log.cc:824] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390: Log is configured to *not* fsync() on all Append() calls
I20240501 02:07:50.031800 27762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390: No bootstrap required, opened a new log
I20240501 02:07:50.033165 27762 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb033f1625cf412897f6cad52cb95390" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39575 } }
I20240501 02:07:50.033288 27762 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:07:50.033311 27762 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb033f1625cf412897f6cad52cb95390, State: Initialized, Role: FOLLOWER
I20240501 02:07:50.033442 27762 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb033f1625cf412897f6cad52cb95390" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39575 } }
I20240501 02:07:50.033524 27762 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:07:50.033556 27762 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:07:50.033639 27762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:07:50.033878 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:50.029710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.22.12.126:39575)
0501 02:07:50.029919 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:50.029926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:50.029944 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:07:50.030376 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:50.030381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:50.030434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:50.030646 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:07:50.031014 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.031767 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:50.032380 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.032621 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:50.033137 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:07:50.033144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:07:50.033145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:07:50.033285 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:07:50.033312 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:07:50.033318 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:50.033377 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:50.033480 (+ 103us) server_negotiation.cc:299] Negotiation successful
0501 02:07:50.033530 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20240501 02:07:50.034220 27762 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb033f1625cf412897f6cad52cb95390" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39575 } }
I20240501 02:07:50.034360 27762 leader_election.cc:302] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb033f1625cf412897f6cad52cb95390; no voters:
I20240501 02:07:50.034523 27762 leader_election.cc:288] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:07:50.034693 27768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:07:50.034780 27762 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:07:50.034776 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:07:50.034845 27768 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [term 1 LEADER]: Becoming Leader. State: Replica: cb033f1625cf412897f6cad52cb95390, State: Running, Role: LEADER
I20240501 02:07:50.034955 27768 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb033f1625cf412897f6cad52cb95390" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39575 } }
I20240501 02:07:50.035244 27768 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb033f1625cf412897f6cad52cb95390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb033f1625cf412897f6cad52cb95390" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39575 } } }
I20240501 02:07:50.035323 27768 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:50.035434 27768 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb033f1625cf412897f6cad52cb95390. Latest consensus state: current_term: 1 leader_uuid: "cb033f1625cf412897f6cad52cb95390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb033f1625cf412897f6cad52cb95390" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39575 } } }
I20240501 02:07:50.035477 27768 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390 [sys.catalog]: This master's current role is: LEADER
I20240501 02:07:50.035593 27775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:07:50.036103 27775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:07:50.037497 27775 catalog_manager.cc:1347] Generated new cluster ID: 9569a6690eca435eaa0c15bbd0bdb36f
I20240501 02:07:50.037544 27775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:07:50.059363 27775 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:07:50.059859 27775 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:07:50.068746 27775 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cb033f1625cf412897f6cad52cb95390: Generated new TSK 0
I20240501 02:07:50.068964 27775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:07:50.095093 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:50.077920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.22.12.126:39575)
0501 02:07:50.077971 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:50.077973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:50.078122 (+ 149us) server_negotiation.cc:407] Connection header received
0501 02:07:50.078265 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:50.078267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:50.078279 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:50.078335 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0501 02:07:50.078670 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.079180 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:50.079778 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.079927 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:50.079969 (+ 42us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 02:07:50.094810 (+ 14841us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0501 02:07:50.094879 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 02:07:50.094969 (+ 90us) server_negotiation.cc:299] Negotiation successful
0501 02:07:50.095001 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28,"thread_start_us":109,"threads_started":1}
I20240501 02:07:50.102350 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:07:50.099617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.22.12.126:39575)
0501 02:07:50.099663 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:07:50.099665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:07:50.099728 (+ 63us) server_negotiation.cc:407] Connection header received
0501 02:07:50.099771 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:07:50.099772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:07:50.099782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:07:50.099848 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0501 02:07:50.100067 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.100870 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:50.101502 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.101694 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:50.101720 (+ 26us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 02:07:50.102137 (+ 417us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0501 02:07:50.102282 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38358: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":25}
W20240501 02:07:50.102424 27765 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38358: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20240501 02:07:50.102497 27798 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:07:50.099545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39575 (local address 127.0.0.1:38358)
0501 02:07:50.099662 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:50.099672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:07:50.099728 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:07:50.099851 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:07:50.099852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:07:50.099854 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0501 02:07:50.100014 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:50.100016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.100953 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:50.100954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:07:50.101425 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:07:50.101428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:50.101486 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:50.101491 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 02:07:50.102247 (+ 756us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0501 02:07:50.102327 (+ 80us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.12.126:39575: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":73,"threads_started":1}
I20240501 02:08:05.108537 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.106575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.22.12.126:39575)
0501 02:08:05.106684 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.106686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.106708 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:08:05.106782 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.106783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.106793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.106861 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0501 02:08:05.107079 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.107572 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.108124 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.108225 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.108240 (+ 15us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 02:08:05.108406 (+ 166us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0501 02:08:05.108484 (+ 78us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59552: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
W20240501 02:08:05.108597 27804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59552: JWT verification failed: token expired
W20240501 02:08:05.108595 27803 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:05.106477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:39575 (local address 127.0.0.1:59552)
0501 02:08:05.106657 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:05.106666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:05.106723 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:05.106870 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:05.106870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:05.106872 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0501 02:08:05.107029 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.107031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.107633 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.107634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:05.108064 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.108067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.108120 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.108126 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 02:08:05.108459 (+ 333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0501 02:08:05.108530 (+ 71us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.12.126:39575: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
W20240501 02:08:05.109917 27804 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:05.109529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.22.12.126:39575)
0501 02:08:05.109578 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.109579 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.109632 (+ 53us) server_negotiation.cc:407] Connection header received
0501 02:08:05.109694 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.109695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.109706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.109756 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:05.109870 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59560: BlockingRecv error: recv got EOF from 127.0.0.1:59560 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20240501 02:08:05.112454 27814 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.22.12.126:39575
W20240501 02:08:05.112984 27804 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:05.112292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.22.12.126:39575)
0501 02:08:05.112344 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.112346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.112439 (+ 93us) server_negotiation.cc:407] Connection header received
0501 02:08:05.112725 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.112726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.112736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.112787 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:05.112945 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59566: BlockingRecv error: recv got EOF from 127.0.0.1:59566 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:08:05.113039 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27695
2024-05-01T02:08:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20240501 02:08:05.151018 22577 mini_oidc.cc:152] Starting JWKS server
I20240501 02:08:05.151764 22577 webserver.cc:460] Webserver started at https://127.0.0.1:38629/ using document root <none> and password file <none>
I20240501 02:08:05.151809 22577 mini_oidc.cc:189] Starting OIDC Discovery server
I20240501 02:08:05.151944 22577 webserver.cc:460] Webserver started at http://127.0.0.1:32769/ using document root <none> and password file <none>
2024-05-01T02:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:05Z Disabled control of system clock
I20240501 02:08:05.158257 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:37817
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:37817
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/testchainca.cert with env {}
W20240501 02:08:05.239346 27820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:05.239545 27820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:05.239567 27820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:05.240924 27820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:05.240974 27820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:05.240985 27820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:05.240998 27820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:05.241020 27820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 02:08:05.241036 27820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 02:08:05.241056 27820 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38629/jwks.json
I20240501 02:08:05.242857 27820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:37817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:37817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:05.243086 27820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:05.243296 27820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:05.247833 27825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:05.248173 27826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:05.248724 27820 server_base.cc:1034] running on GCE node
W20240501 02:08:05.249023 27828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:05.249285 27820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:05.249516 27820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:05.250653 27820 hybrid_clock.cc:648] HybridClock initialized: now 1714529285250633 us; error 36 us; skew 500 ppm
I20240501 02:08:05.251936 27820 webserver.cc:460] Webserver started at http://127.22.12.126:37825/ using document root <none> and password file <none>
I20240501 02:08:05.252113 27820 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:05.252152 27820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:05.252239 27820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:05.253261 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "5cafd6a5783349f59c043bb8217a0aaf"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "ba701e7ce6f1aad619a634db68647e2b"
server_key_iv: "eb656bb4ffa9a73144a44b17fcd02ecd"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.253568 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "5cafd6a5783349f59c043bb8217a0aaf"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "ba701e7ce6f1aad619a634db68647e2b"
server_key_iv: "eb656bb4ffa9a73144a44b17fcd02ecd"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.254760 27820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:08:05.255369 27834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:05.255527 27820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:05.255600 27820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "5cafd6a5783349f59c043bb8217a0aaf"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "ba701e7ce6f1aad619a634db68647e2b"
server_key_iv: "eb656bb4ffa9a73144a44b17fcd02ecd"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.255659 27820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:05.272419 27820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:05.272702 27820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:05.272810 27820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:05.276768 27820 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:37817
I20240501 02:08:05.276796 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:37817 every 8 connection(s)
I20240501 02:08:05.277122 27820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:05.277750 27887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:05.280304 27887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf: Bootstrap starting.
I20240501 02:08:05.280848 27887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:05.281085 27887 log.cc:824] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:05.281741 27887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf: No bootstrap required, opened a new log
I20240501 02:08:05.281986 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27820
I20240501 02:08:05.282076 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:05.282225 22577 external_mini_cluster.cc:1428] Setting key 905a3456ccdb80fc338c1ef1424e5401
I20240501 02:08:05.283576 27887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cafd6a5783349f59c043bb8217a0aaf" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37817 } }
I20240501 02:08:05.283751 27887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:05.283780 27887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cafd6a5783349f59c043bb8217a0aaf, State: Initialized, Role: FOLLOWER
I20240501 02:08:05.283929 27887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cafd6a5783349f59c043bb8217a0aaf" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37817 } }
I20240501 02:08:05.283995 27887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:05.284015 27887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:05.284049 27887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:05.284817 27887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cafd6a5783349f59c043bb8217a0aaf" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37817 } }
I20240501 02:08:05.284915 27887 leader_election.cc:302] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cafd6a5783349f59c043bb8217a0aaf; no voters:
I20240501 02:08:05.285063 27887 leader_election.cc:288] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:05.285111 27892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:05.285291 27892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [term 1 LEADER]: Becoming Leader. State: Replica: 5cafd6a5783349f59c043bb8217a0aaf, State: Running, Role: LEADER
I20240501 02:08:05.285308 27887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:05.285410 27892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cafd6a5783349f59c043bb8217a0aaf" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37817 } }
I20240501 02:08:05.285715 27892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cafd6a5783349f59c043bb8217a0aaf. Latest consensus state: current_term: 1 leader_uuid: "5cafd6a5783349f59c043bb8217a0aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cafd6a5783349f59c043bb8217a0aaf" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37817 } } }
I20240501 02:08:05.285727 27893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cafd6a5783349f59c043bb8217a0aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cafd6a5783349f59c043bb8217a0aaf" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37817 } } }
I20240501 02:08:05.285879 27893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:05.286062 27892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:05.286222 27899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:05.286726 27899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:05.287628 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.282718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.22.12.126:37817)
0501 02:08:05.282946 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.282955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.282977 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:08:05.283544 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.283553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.283624 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.283785 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:05.284203 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.284986 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.285749 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.285981 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.286929 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:05.286938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:05.286938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:08:05.287099 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:05.287138 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:08:05.287148 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:05.287194 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:05.287409 (+ 215us) server_negotiation.cc:299] Negotiation successful
0501 02:08:05.287466 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20240501 02:08:05.288471 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:08:05.288493 27899 catalog_manager.cc:1347] Generated new cluster ID: 7e94d464fe54449daa993d8558271cc3
I20240501 02:08:05.288534 27899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:08:05.295148 27899 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:05.296082 27899 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:05.301333 27899 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5cafd6a5783349f59c043bb8217a0aaf: Generated new TSK 0
I20240501 02:08:05.301533 27899 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240501 02:08:05.328372 27917 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240501 02:08:05.328467 27917 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240501 02:08:05.328749 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.306199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.22.12.126:37817)
0501 02:08:05.306248 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.306251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.306360 (+ 109us) server_negotiation.cc:407] Connection header received
0501 02:08:05.306447 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.306448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.306460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.306524 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0501 02:08:05.306798 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.307305 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.307933 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.308078 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.308127 (+ 49us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 02:08:05.328525 (+ 20398us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:08:05.328682 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20240501 02:08:05.328805 27891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20240501 02:08:05.328861 27916 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:05.306139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:37817 (local address 127.0.0.1:34074)
0501 02:08:05.306289 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:05.306299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:05.306369 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:05.306552 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:05.306554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:05.306556 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0501 02:08:05.306733 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.306736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.307409 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.307410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:05.307850 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.307852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.307915 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.307922 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 02:08:05.328638 (+ 20716us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:08:05.328761 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.12.126:37817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20240501 02:08:05.329217 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27820
2024-05-01T02:08:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20240501 02:08:05.366071 22577 mini_oidc.cc:152] Starting JWKS server
I20240501 02:08:05.366878 22577 webserver.cc:460] Webserver started at https://127.0.0.1:43135/ using document root <none> and password file <none>
I20240501 02:08:05.366926 22577 mini_oidc.cc:189] Starting OIDC Discovery server
I20240501 02:08:05.367033 22577 webserver.cc:460] Webserver started at http://127.0.0.1:44563/ using document root <none> and password file <none>
2024-05-01T02:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:05Z Disabled control of system clock
I20240501 02:08:05.373797 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:32937
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:32937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43135/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240501 02:08:05.452980 27923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:05.453181 27923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:05.453213 27923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:05.454586 27923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:05.454629 27923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:05.454646 27923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:05.454661 27923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:05.454676 27923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 02:08:05.454699 27923 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 02:08:05.454721 27923 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43135/jwks.json
I20240501 02:08:05.456223 27923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:32937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43135/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:05.456457 27923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:05.456697 27923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:05.460732 27929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:05.461143 27931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:05.461568 27928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:05.462049 27923 server_base.cc:1034] running on GCE node
I20240501 02:08:05.462250 27923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:05.462503 27923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:05.463660 27923 hybrid_clock.cc:648] HybridClock initialized: now 1714529285463643 us; error 32 us; skew 500 ppm
I20240501 02:08:05.464866 27923 webserver.cc:460] Webserver started at http://127.22.12.126:35201/ using document root <none> and password file <none>
I20240501 02:08:05.465042 27923 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:05.465083 27923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:05.465189 27923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:05.466164 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "9546216ae56d75b2cb8422e7ad964dca"
server_key_iv: "a6661301aa9b5ba458f8a56991370b84"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.466495 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "9546216ae56d75b2cb8422e7ad964dca"
server_key_iv: "a6661301aa9b5ba458f8a56991370b84"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.467621 27923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:05.468228 27937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:05.468394 27923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:05.468466 27923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "9546216ae56d75b2cb8422e7ad964dca"
server_key_iv: "a6661301aa9b5ba458f8a56991370b84"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.468531 27923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:05.494122 27923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:05.494448 27923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:05.494598 27923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:05.498277 27923 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:32937
I20240501 02:08:05.498346 27989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:32937 every 8 connection(s)
I20240501 02:08:05.498651 27923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:05.499222 27990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:05.501699 27990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc: Bootstrap starting.
I20240501 02:08:05.502207 27990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:05.502435 27990 log.cc:824] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:05.503176 27990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc: No bootstrap required, opened a new log
I20240501 02:08:05.504608 27990 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32937 } }
I20240501 02:08:05.504737 27990 raft_consensus.cc:374] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:05.504755 27990 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03e16dbf63a4ebd9c684d8b6c0c61dc, State: Initialized, Role: FOLLOWER
I20240501 02:08:05.504877 27990 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32937 } }
I20240501 02:08:05.504936 27990 raft_consensus.cc:388] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:05.504958 27990 raft_consensus.cc:482] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:05.505002 27990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:05.505575 27990 raft_consensus.cc:504] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32937 } }
I20240501 02:08:05.505668 27990 leader_election.cc:302] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d03e16dbf63a4ebd9c684d8b6c0c61dc; no voters:
I20240501 02:08:05.505817 27990 leader_election.cc:288] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:05.505878 27993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:05.506029 27993 raft_consensus.cc:686] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [term 1 LEADER]: Becoming Leader. State: Replica: d03e16dbf63a4ebd9c684d8b6c0c61dc, State: Running, Role: LEADER
I20240501 02:08:05.506047 27990 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:05.506114 27993 consensus_queue.cc:237] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32937 } }
I20240501 02:08:05.506441 27994 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32937 } } }
I20240501 02:08:05.506618 27994 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:05.506451 27995 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03e16dbf63a4ebd9c684d8b6c0c61dc. Latest consensus state: current_term: 1 leader_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03e16dbf63a4ebd9c684d8b6c0c61dc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32937 } } }
I20240501 02:08:05.506918 27995 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:05.507011 28002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:05.507474 28002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:05.507902 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 27923
I20240501 02:08:05.507999 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:05.508173 22577 external_mini_cluster.cc:1428] Setting key bf6c0b40cf475f98e1ae08cd87bc67e0
I20240501 02:08:05.509341 28002 catalog_manager.cc:1347] Generated new cluster ID: 6804e8688f7743ca9c0ea874581f8231
I20240501 02:08:05.509397 28002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:08:05.514153 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.508561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.22.12.126:32937)
0501 02:08:05.508746 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.508753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.508928 (+ 175us) server_negotiation.cc:407] Connection header received
0501 02:08:05.509043 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.509049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.509095 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.509219 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:05.510957 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.512092 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.512890 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.513112 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.513532 (+ 420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:05.513538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:05.513539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:08:05.513743 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:05.513772 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:08:05.513780 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:05.513844 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:05.513922 (+ 78us) server_negotiation.cc:299] Negotiation successful
0501 02:08:05.513979 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20240501 02:08:05.515159 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:08:05.538372 28002 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:05.539183 28002 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:05.549649 28002 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d03e16dbf63a4ebd9c684d8b6c0c61dc: Generated new TSK 0
I20240501 02:08:05.549872 28002 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240501 02:08:05.585044 28020 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240501 02:08:05.585137 28020 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240501 02:08:05.586160 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.561457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.22.12.126:32937)
0501 02:08:05.561505 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.561507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.561579 (+ 72us) server_negotiation.cc:407] Connection header received
0501 02:08:05.561667 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.561668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.561678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.561734 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0501 02:08:05.562046 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.562582 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.563412 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.563536 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.563581 (+ 45us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 02:08:05.585927 (+ 22346us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:08:05.586104 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20240501 02:08:05.586212 28013 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20240501 02:08:05.586248 28019 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:05.561351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:32937 (local address 127.0.0.1:53648)
0501 02:08:05.561502 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:05.561512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:05.561583 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:05.561745 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:05.561746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:05.561748 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0501 02:08:05.561986 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.561988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.562663 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.562664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:05.563331 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.563333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.563418 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.563424 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 02:08:05.586027 (+ 22603us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:08:05.586155 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.12.126:32937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20240501 02:08:05.586675 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 27923
2024-05-01T02:08:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2024-05-01T02:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:05Z Disabled control of system clock
I20240501 02:08:05.631111 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:43543
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:43543
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/testchainca.cert with env {}
W20240501 02:08:05.709965 28024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:05.710170 28024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:05.710201 28024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:05.711622 28024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:05.711674 28024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:05.711690 28024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:05.711706 28024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:05.711719 28024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 02:08:05.711735 28024 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 02:08:05.711755 28024 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20240501 02:08:05.713510 28024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:43543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:43543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:05.713762 28024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:05.713939 28024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:05.718358 28032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:05.718842 28030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:05.719209 28029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:05.719818 28024 server_base.cc:1034] running on GCE node
I20240501 02:08:05.720078 28024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:05.720346 28024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:05.721498 28024 hybrid_clock.cc:648] HybridClock initialized: now 1714529285721484 us; error 41 us; skew 500 ppm
I20240501 02:08:05.722909 28024 webserver.cc:460] Webserver started at http://127.22.12.126:41567/ using document root <none> and password file <none>
I20240501 02:08:05.723109 28024 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:05.723166 28024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:05.723294 28024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:05.724319 28024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "82dd930a7760373ff2d54a3e58662bdc"
server_key_iv: "1319410f54723c3d2aaeed60a431c102"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.724642 28024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "82dd930a7760373ff2d54a3e58662bdc"
server_key_iv: "1319410f54723c3d2aaeed60a431c102"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.725831 28024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:08:05.726605 28038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:05.726806 28024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:05.726900 28024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1"
format_stamp: "Formatted at 2024-05-01 02:08:05 on dist-test-slave-9tkh"
server_key: "82dd930a7760373ff2d54a3e58662bdc"
server_key_iv: "1319410f54723c3d2aaeed60a431c102"
server_key_version: "encryptionkey@0"
I20240501 02:08:05.726962 28024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:05.740834 28024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:05.741130 28024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:05.741240 28024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:05.744982 28024 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:43543
I20240501 02:08:05.745046 28090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:43543 every 8 connection(s)
I20240501 02:08:05.745347 28024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:05.745935 28091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:05.748389 28091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1: Bootstrap starting.
I20240501 02:08:05.748930 28091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:05.749167 28091 log.cc:824] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:05.749835 28091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1: No bootstrap required, opened a new log
I20240501 02:08:05.751308 28091 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 43543 } }
I20240501 02:08:05.751436 28091 raft_consensus.cc:374] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:05.751451 28091 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe1c69ef5b9457eb82f1ea5d2dd27d1, State: Initialized, Role: FOLLOWER
I20240501 02:08:05.751570 28091 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 43543 } }
I20240501 02:08:05.751627 28091 raft_consensus.cc:388] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:05.751648 28091 raft_consensus.cc:482] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:05.751689 28091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:05.752238 28091 raft_consensus.cc:504] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 43543 } }
I20240501 02:08:05.752321 28091 leader_election.cc:302] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe1c69ef5b9457eb82f1ea5d2dd27d1; no voters:
I20240501 02:08:05.752461 28091 leader_election.cc:288] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:05.752525 28094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:05.752671 28094 raft_consensus.cc:686] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [term 1 LEADER]: Becoming Leader. State: Replica: afe1c69ef5b9457eb82f1ea5d2dd27d1, State: Running, Role: LEADER
I20240501 02:08:05.752702 28091 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:05.752759 28094 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 43543 } }
I20240501 02:08:05.753095 28095 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 43543 } } }
I20240501 02:08:05.753291 28095 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:05.753115 28096 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe1c69ef5b9457eb82f1ea5d2dd27d1. Latest consensus state: current_term: 1 leader_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe1c69ef5b9457eb82f1ea5d2dd27d1" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 43543 } } }
I20240501 02:08:05.753662 28096 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:05.753875 28103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:05.754330 28103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:05.755157 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28024
I20240501 02:08:05.755246 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:05.755391 22577 external_mini_cluster.cc:1428] Setting key a8f7b9205d4a1d15d8ff6014724c01f6
I20240501 02:08:05.755687 28103 catalog_manager.cc:1347] Generated new cluster ID: 7ae199da381c4cc5a949bbd89747fd39
I20240501 02:08:05.755738 28103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:08:05.758647 28103 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:05.759110 28103 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:05.770567 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.755984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.22.12.126:43543)
0501 02:08:05.758272 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.758278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.758296 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:08:05.758382 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.758388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.758447 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.766918 (+ 8471us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:05.767072 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.768316 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.769235 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.769464 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.769917 (+ 453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:05.769926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:05.769927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:08:05.770113 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:05.770149 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:08:05.770159 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:05.770226 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:05.770306 (+ 80us) server_negotiation.cc:299] Negotiation successful
0501 02:08:05.770365 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2227,"thread_start_us":63,"threads_started":1}
I20240501 02:08:05.771447 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:08:05.775000 28103 catalog_manager.cc:5857] T 00000000000000000000000000000000 P afe1c69ef5b9457eb82f1ea5d2dd27d1: Generated new TSK 0
I20240501 02:08:05.775203 28103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:08:05.824433 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:05.781450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.22.12.126:43543)
0501 02:08:05.781498 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:05.781500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:05.781570 (+ 70us) server_negotiation.cc:407] Connection header received
0501 02:08:05.781665 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:05.781666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:05.781688 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:05.781758 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0501 02:08:05.782084 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.782623 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.783481 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.783624 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.783673 (+ 49us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 02:08:05.824205 (+ 40532us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0501 02:08:05.824371 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56466: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20240501 02:08:05.824497 28114 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56466: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20240501 02:08:05.824590 28120 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:05.781350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:43543 (local address 127.0.0.1:56466)
0501 02:08:05.781494 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:05.781505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:05.781578 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:05.781769 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:05.781770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:05.781772 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0501 02:08:05.782017 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.782019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.782693 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:05.782694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:05.783373 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:05.783377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:05.783460 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:05.783466 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 02:08:05.824343 (+ 40877us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0501 02:08:05.824494 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.12.126:43543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20240501 02:08:05.824914 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28024
2024-05-01T02:08:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (217 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:08:06 dist-test-slave-9tkh krb5kdc[28132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:08:06 dist-test-slave-9tkh krb5kdc[28132](info): set up 2 sockets
May 01 02:08:06 dist-test-slave-9tkh krb5kdc[28132](info): commencing operation
krb5kdc: starting...
W20240501 02:08:06.894656 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.880s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:08:06 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:08:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:08:07.009511 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:46115
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:46115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:07.087915 28145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:07.088121 28145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:07.088153 28145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:07.089521 28145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:07.089569 28145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:07.089586 28145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:07.089602 28145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:07.089615 28145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:07.091336 28145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:46115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:46115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:07.091573 28145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:07.091758 28145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:07.095942 28150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.096350 28153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.096805 28151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:07.097200 28145 server_base.cc:1034] running on GCE node
I20240501 02:08:07.097409 28145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:07.097638 28145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:07.098764 28145 hybrid_clock.cc:648] HybridClock initialized: now 1714529287098754 us; error 23 us; skew 500 ppm
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.101186 28145 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:08:07.101600 28145 webserver.cc:460] Webserver started at http://127.22.12.126:37371/ using document root <none> and password file <none>
I20240501 02:08:07.101763 28145 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:07.101811 28145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:07.101917 28145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:07.102945 28145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "aaa2c12e899d4b00b4177176c860adee"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "4eea4b0d22282aabd9d7c3aa3da83c8d"
server_key_iv: "81ff599cb52a3f8df4689be740b61b00"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.103266 28145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "aaa2c12e899d4b00b4177176c860adee"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "4eea4b0d22282aabd9d7c3aa3da83c8d"
server_key_iv: "81ff599cb52a3f8df4689be740b61b00"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.104389 28145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:08:07.104992 28160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.105154 28145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:07.105229 28145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "aaa2c12e899d4b00b4177176c860adee"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "4eea4b0d22282aabd9d7c3aa3da83c8d"
server_key_iv: "81ff599cb52a3f8df4689be740b61b00"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.105294 28145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:07.137562 28145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:07.139709 28145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:07.139829 28145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:07.143677 28145 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:46115
I20240501 02:08:07.143728 28212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:46115 every 8 connection(s)
I20240501 02:08:07.144032 28145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:07.144646 28213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:07.147189 28213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee: Bootstrap starting.
I20240501 02:08:07.147728 28213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:07.147964 28213 log.cc:824] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:07.148597 28213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee: No bootstrap required, opened a new log
I20240501 02:08:07.150065 28213 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2c12e899d4b00b4177176c860adee" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46115 } }
I20240501 02:08:07.150194 28213 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:07.150218 28213 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa2c12e899d4b00b4177176c860adee, State: Initialized, Role: FOLLOWER
I20240501 02:08:07.150348 28213 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2c12e899d4b00b4177176c860adee" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46115 } }
I20240501 02:08:07.150406 28213 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:07.150431 28213 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:07.150480 28213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:07.151042 28213 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2c12e899d4b00b4177176c860adee" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46115 } }
I20240501 02:08:07.151137 28213 leader_election.cc:302] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaa2c12e899d4b00b4177176c860adee; no voters:
I20240501 02:08:07.151298 28213 leader_election.cc:288] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:07.151360 28216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:07.151540 28213 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:07.151542 28216 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [term 1 LEADER]: Becoming Leader. State: Replica: aaa2c12e899d4b00b4177176c860adee, State: Running, Role: LEADER
I20240501 02:08:07.151715 28216 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2c12e899d4b00b4177176c860adee" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46115 } }
I20240501 02:08:07.152065 28217 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaa2c12e899d4b00b4177176c860adee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2c12e899d4b00b4177176c860adee" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46115 } } }
I20240501 02:08:07.152180 28218 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa2c12e899d4b00b4177176c860adee. Latest consensus state: current_term: 1 leader_uuid: "aaa2c12e899d4b00b4177176c860adee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa2c12e899d4b00b4177176c860adee" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46115 } } }
I20240501 02:08:07.152202 28217 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:07.152264 28218 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee [sys.catalog]: This master's current role is: LEADER
W20240501 02:08:07.153466 28232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:08:07.153509 28232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:08:07.153559 28225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:07.153717 28225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:07.154089 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28145
I20240501 02:08:07.154182 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:07.154345 22577 external_mini_cluster.cc:1428] Setting key 64c0612708020081f3fde980178216a7
I20240501 02:08:07.155158 28225 catalog_manager.cc:1347] Generated new cluster ID: 9e26e1c21a7047f3a12a3f4e11eecc86
I20240501 02:08:07.155211 28225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:07.168269 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.154887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.22.12.126:46115)
0501 02:08:07.155008 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.155018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.155042 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:08:07.158655 (+ 3613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.158663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.158727 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.158890 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.159767 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.161349 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.162098 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.162369 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.164862 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.164871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.164871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.164895 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.166868 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.167239 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.167241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.167241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.167294 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.167436 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.167437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.167437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.167678 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.167799 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.167936 (+ 137us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.168057 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20240501 02:08:07.174757 28225 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:07.175336 28225 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:07.179885 28225 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aaa2c12e899d4b00b4177176c860adee: Generated new TSK 0
I20240501 02:08:07.180110 28225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:08:07.212939 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46115
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:07.293035 28239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:07.293246 28239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:07.293272 28239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:07.294692 28239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:07.294744 28239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:07.294768 28239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:08:07.296474 28239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:07.296702 28239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:07.296902 28239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:07.302093 28247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.302718 28245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.303217 28244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:07.303719 28239 server_base.cc:1034] running on GCE node
I20240501 02:08:07.303937 28239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:07.304179 28239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:07.305367 28239 hybrid_clock.cc:648] HybridClock initialized: now 1714529287305352 us; error 39 us; skew 500 ppm
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.308141 28239 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:08:07.308662 28239 webserver.cc:460] Webserver started at http://127.22.12.65:34153/ using document root <none> and password file <none>
I20240501 02:08:07.308864 28239 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:07.308913 28239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:07.309002 28239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:07.309960 28239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "7daddf5c7fe64fb2ba68aa56c5779851"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "747ce7301786aebb8930d524829769f8"
server_key_iv: "be18ec54cd9a86a0f559f8432142639e"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.310220 28239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "7daddf5c7fe64fb2ba68aa56c5779851"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "747ce7301786aebb8930d524829769f8"
server_key_iv: "be18ec54cd9a86a0f559f8432142639e"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.311422 28239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:08:07.312047 28254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.312202 28239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:07.312274 28239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "7daddf5c7fe64fb2ba68aa56c5779851"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "747ce7301786aebb8930d524829769f8"
server_key_iv: "be18ec54cd9a86a0f559f8432142639e"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.312356 28239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:07.343052 28239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:07.345217 28239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:07.345336 28239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:07.345551 28239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:07.345844 28239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:07.345875 28239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.345903 28239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:07.345928 28239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.351156 28239 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:37985
I20240501 02:08:07.351202 28367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:37985 every 8 connection(s)
I20240501 02:08:07.351508 28239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:07.357724 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28239
I20240501 02:08:07.357820 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:08:07.357985 22577 external_mini_cluster.cc:1428] Setting key 5e56cd1a3dac8491a31aff0ea8bd43d2
I20240501 02:08:07.360280 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.352273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:37625 (local address 127.22.12.126:46115)
0501 02:08:07.352323 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.352325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.352456 (+ 131us) server_negotiation.cc:407] Connection header received
0501 02:08:07.352904 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.352905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.352922 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.352974 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.353773 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.354263 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.354888 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.354997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.357353 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.357358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.357358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.357365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.358841 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.359289 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.359291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.359291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.359318 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.359552 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.359553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.359553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.359651 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.359706 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.360061 (+ 355us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.360181 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:08:07.360302 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.352181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:46115 (local address 127.22.12.65:37625)
0501 02:08:07.352336 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:07.352363 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:07.352815 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:07.353018 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:07.353022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:07.353378 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:07.353696 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:07.353701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.354348 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.354349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:07.354817 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:07.354819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.355210 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.355759 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:07.355776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:07.357252 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:07.358918 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:07.358926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:07.358929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:07.359195 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:07.359386 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:07.359387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:07.359387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:07.359467 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:07.359705 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:07.359710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:07.359864 (+ 154us) client_negotiation.cc:769] Sending connection context
0501 02:08:07.359991 (+ 127us) client_negotiation.cc:240] Negotiation successful
0501 02:08:07.360154 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20240501 02:08:07.360811 28368 heartbeater.cc:347] Connected to a master server at 127.22.12.126:46115
I20240501 02:08:07.360906 28368 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:07.361117 28368 heartbeater.cc:511] Master 127.22.12.126:46115 requested a full tablet report, sending...
I20240501 02:08:07.361735 28177 ts_manager.cc:194] Registered new tserver with Master: 7daddf5c7fe64fb2ba68aa56c5779851 (127.22.12.65:37985)
I20240501 02:08:07.362741 28177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:37625
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:08:07.382850 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46115
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:07.461748 28373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:07.461925 28373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:07.461956 28373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:07.463315 28373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:07.463357 28373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:07.463390 28373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:08:07.464972 28373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:07.465185 28373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:07.465359 28373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:07.470770 28379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.471064 28378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.471593 28381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:07.471613 28373 server_base.cc:1034] running on GCE node
I20240501 02:08:07.472115 28373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:07.472322 28373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:07.473471 28373 hybrid_clock.cc:648] HybridClock initialized: now 1714529287473439 us; error 51 us; skew 500 ppm
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.476202 28373 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:08:07.476675 28373 webserver.cc:460] Webserver started at http://127.22.12.66:33109/ using document root <none> and password file <none>
I20240501 02:08:07.476867 28373 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:07.476912 28373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:07.477015 28373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:07.478015 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "6c393a3965a34e34a656081a6315c277"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "9ba26875b74df48055a7d004f69a7946"
server_key_iv: "050a5fca7662fa2ba93b0442c357babb"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.478307 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "6c393a3965a34e34a656081a6315c277"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "9ba26875b74df48055a7d004f69a7946"
server_key_iv: "050a5fca7662fa2ba93b0442c357babb"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.479583 28373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:08:07.480441 28388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.480666 28373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:07.480753 28373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "6c393a3965a34e34a656081a6315c277"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "9ba26875b74df48055a7d004f69a7946"
server_key_iv: "050a5fca7662fa2ba93b0442c357babb"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.480823 28373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:07.504315 28373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:07.506515 28373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:07.506682 28373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:07.506901 28373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:07.507203 28373 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:07.507236 28373 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.507265 28373 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:07.507282 28373 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.512715 28373 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:33657
I20240501 02:08:07.512756 28501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:33657 every 8 connection(s)
I20240501 02:08:07.513058 28373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:08:07.517398 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28373
I20240501 02:08:07.517513 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:08:07.517699 22577 external_mini_cluster.cc:1428] Setting key b188425f9d67deaa7f8dfa2edcb0536c
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:07.522063 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.513818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:40789 (local address 127.22.12.126:46115)
0501 02:08:07.513886 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.513888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.514032 (+ 144us) server_negotiation.cc:407] Connection header received
0501 02:08:07.514587 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.514588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.514598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.514654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.515473 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.515992 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.516562 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.516666 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.519357 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.519361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.519361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.519368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.520797 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.521230 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.521231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.521231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.521253 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.521452 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.521453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.521453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.521543 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.521595 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.521881 (+ 286us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.521970 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:08:07.522181 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.513765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:46115 (local address 127.22.12.66:40789)
0501 02:08:07.513920 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:07.513948 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:07.514426 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:07.514733 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:07.514736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:07.515040 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:07.515368 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:07.515373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.516074 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.516075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:07.516496 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:07.516498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.516867 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.517458 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:07.517487 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:07.519208 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:07.520886 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:07.520893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:07.520896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:07.521157 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:07.521309 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:07.521310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:07.521310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:07.521396 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:07.521611 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:07.521617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:07.521743 (+ 126us) client_negotiation.cc:769] Sending connection context
0501 02:08:07.521858 (+ 115us) client_negotiation.cc:240] Negotiation successful
0501 02:08:07.521998 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20240501 02:08:07.522651 28502 heartbeater.cc:347] Connected to a master server at 127.22.12.126:46115
I20240501 02:08:07.522766 28502 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:07.523039 28502 heartbeater.cc:511] Master 127.22.12.126:46115 requested a full tablet report, sending...
I20240501 02:08:07.523473 28177 ts_manager.cc:194] Registered new tserver with Master: 6c393a3965a34e34a656081a6315c277 (127.22.12.66:33657)
I20240501 02:08:07.523854 28177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:40789
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:08:07.543572 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46115
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:07.623278 28507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:07.623471 28507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:07.623495 28507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:07.624863 28507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:07.624915 28507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:07.624940 28507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:08:07.626626 28507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:07.626850 28507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:07.627033 28507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:07.631558 28513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.631821 28512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:07.632160 28515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:07.632622 28507 server_base.cc:1034] running on GCE node
I20240501 02:08:07.632815 28507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:07.633029 28507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:07.634171 28507 hybrid_clock.cc:648] HybridClock initialized: now 1714529287634157 us; error 36 us; skew 500 ppm
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.636531 28507 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:08:07.636894 28507 webserver.cc:460] Webserver started at http://127.22.12.67:37557/ using document root <none> and password file <none>
I20240501 02:08:07.637035 28507 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:07.637068 28507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:07.637147 28507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:07.638079 28507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "653d25d5d54e4796a81b6d7e82da4412"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "f6b7329e0d92e211248bb657efc186ec"
server_key_iv: "79c47b8c50b00543f175f41a7ccc71a0"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.638326 28507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "653d25d5d54e4796a81b6d7e82da4412"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "f6b7329e0d92e211248bb657efc186ec"
server_key_iv: "79c47b8c50b00543f175f41a7ccc71a0"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.639499 28507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 02:08:07.640117 28522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.640280 28507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.640353 28507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "653d25d5d54e4796a81b6d7e82da4412"
format_stamp: "Formatted at 2024-05-01 02:08:07 on dist-test-slave-9tkh"
server_key: "f6b7329e0d92e211248bb657efc186ec"
server_key_iv: "79c47b8c50b00543f175f41a7ccc71a0"
server_key_version: "encryptionkey@0"
I20240501 02:08:07.640408 28507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:07.661545 28507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:07.663570 28507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:07.663668 28507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:07.663836 28507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:07.664103 28507 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:07.664134 28507 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.664156 28507 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:07.664178 28507 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:07.669121 28507 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:42407
I20240501 02:08:07.669185 28635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:42407 every 8 connection(s)
I20240501 02:08:07.669432 28507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:07.677453 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.670096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:53013 (local address 127.22.12.126:46115)
0501 02:08:07.670157 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.670159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.670320 (+ 161us) server_negotiation.cc:407] Connection header received
0501 02:08:07.670808 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.670809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.670819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.670876 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.671600 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.672081 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.672587 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.672688 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.674887 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.674892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.674892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.674899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.676269 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.676656 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.676658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.676658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.676683 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.676847 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.676848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.676849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.676950 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.677004 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.677255 (+ 251us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.677351 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:08:07.677564 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.670062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:46115 (local address 127.22.12.67:53013)
0501 02:08:07.670216 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:07.670243 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:07.670718 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:07.670933 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:07.670937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:07.671225 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:07.671529 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:07.671534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.672150 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.672150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:07.672532 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:07.672533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.672829 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.673324 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:07.673341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:07.674771 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:07.676368 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:07.676375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:07.676379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:07.676585 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:07.676737 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:07.676738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:07.676738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:07.676798 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:07.677001 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:07.677007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:07.677129 (+ 122us) client_negotiation.cc:769] Sending connection context
0501 02:08:07.677227 (+ 98us) client_negotiation.cc:240] Negotiation successful
0501 02:08:07.677372 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20240501 02:08:07.678102 28636 heartbeater.cc:347] Connected to a master server at 127.22.12.126:46115
I20240501 02:08:07.678200 28636 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:07.678277 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28507
I20240501 02:08:07.678359 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:08:07.678433 28636 heartbeater.cc:511] Master 127.22.12.126:46115 requested a full tablet report, sending...
I20240501 02:08:07.678506 22577 external_mini_cluster.cc:1428] Setting key dc9d18b427b8c83b0ea19c7dc5ebacc6
I20240501 02:08:07.678916 28177 ts_manager.cc:194] Registered new tserver with Master: 653d25d5d54e4796a81b6d7e82da4412 (127.22.12.67:42407)
I20240501 02:08:07.679354 28177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:53013
I20240501 02:08:07.679548 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/test-admin.keytab.
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.695001 22577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20240501 02:08:07.696775 28643 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:07.695913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.22.12.65:37985)
0501 02:08:07.696109 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.696116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.696126 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:08:07.696184 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.696186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.696200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.696268 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.696688 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59968: BlockingRecv error: recv got EOF from 127.0.0.1:59968 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
W20240501 02:08:07.696939 22577 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc' of type FILE
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.698057 22577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.698998 22577 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:08:07.705902 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.699606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.22.12.65:37985)
0501 02:08:07.699659 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.699661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.699682 (+ 21us) server_negotiation.cc:407] Connection header received
0501 02:08:07.699748 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.699749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.699759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.699831 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.700298 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.701007 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.701574 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.701782 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.703523 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.703535 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.703535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.703542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.705025 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.705361 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.705363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.705364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.705399 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.705507 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.705507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.705508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.705605 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.705714 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.705753 (+ 39us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.705823 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20240501 02:08:07.707681 28643 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:07.707145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.22.12.65:37985)
0501 02:08:07.707203 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.707205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.707213 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:07.707265 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.707265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.707274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.707319 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.707627 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59980: BlockingRecv error: recv got EOF from 127.0.0.1:59980 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20240501 02:08:07.707777 22577 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc' of type FILE
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.708750 22577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.709630 22577 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:08:07.716069 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.710191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.22.12.65:37985)
0501 02:08:07.710234 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.710236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.710243 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:08:07.710284 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.710285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.710292 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.710339 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.710874 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.711340 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.711865 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.711974 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.713729 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.713733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.713733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.713738 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.715211 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.715481 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.715483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.715483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.715505 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.715628 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.715629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.715630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.715713 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.715795 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.715892 (+ 97us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.715973 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20240501 02:08:07.717592 28643 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:07.717107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.22.12.65:37985)
0501 02:08:07.717168 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.717169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.717178 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:07.717243 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.717243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.717252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.717302 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.717544 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59988: BlockingRecv error: recv got EOF from 127.0.0.1:59988 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20240501 02:08:07.717702 22577 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714529249419714-22577-0/krb5kdc/krb5cc' of type FILE
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.718686 22577 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:07.719491 22577 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.65@KRBTEST.COM
I20240501 02:08:07.725836 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:07.720017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.22.12.65:37985)
0501 02:08:07.720069 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:07.720071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:07.720080 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:07.720139 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:07.720140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:07.720148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:07.720199 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:07.720626 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.721055 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:07.721543 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:07.721712 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:07.723662 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:07.723666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:07.723666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:07.723672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:07.724984 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.725292 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.725293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.725294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.725318 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:07.725448 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:07.725448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:07.725448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:07.725522 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:07.725589 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:07.725669 (+ 80us) server_negotiation.cc:299] Negotiation successful
0501 02:08:07.725737 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:08:07.726522 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28239
I20240501 02:08:07.731267 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28373
I20240501 02:08:07.735208 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28507
I20240501 02:08:07.739607 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28145
2024-05-01T02:08:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1744 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28656](info): set up 2 sockets
May 01 02:08:07 dist-test-slave-9tkh krb5kdc[28656](info): commencing operation
krb5kdc: starting...
W20240501 02:08:08.630471 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:08:08 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab.
Entry for principal oryx/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab.
I20240501 02:08:08.739667 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35589
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab
--principal=oryx/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:08.816586 28669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:08.816789 28669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:08.816820 28669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:08.818105 28669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:08.818152 28669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:08.818169 28669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:08.818184 28669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:08.818198 28669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:08.819727 28669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab
--principal=oryx/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:08.819954 28669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:08.820122 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:08.824373 28675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:08.824808 28674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:08.825198 28677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:08.825620 28669 server_base.cc:1034] running on GCE node
I20240501 02:08:08.825853 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:08.826122 28669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:08.827260 28669 hybrid_clock.cc:648] HybridClock initialized: now 1714529288827250 us; error 28 us; skew 500 ppm
May 01 02:08:08 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:08.829599 28669 init.cc:377] Logged in from keytab as oryx/127.22.12.126@KRBTEST.COM (short username oryx)
I20240501 02:08:08.829989 28669 webserver.cc:460] Webserver started at http://127.22.12.126:41825/ using document root <none> and password file <none>
I20240501 02:08:08.830164 28669 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:08.830204 28669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:08.830307 28669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:08.831300 28669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "34b24cf1f77b46c6a5ff8821786121c7"
format_stamp: "Formatted at 2024-05-01 02:08:08 on dist-test-slave-9tkh"
server_key: "1c9fef90c6b313f673d5d1047d79ccaf"
server_key_iv: "42b53d3388fde095a9c518dc1f1e5398"
server_key_version: "encryptionkey@0"
I20240501 02:08:08.831602 28669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "34b24cf1f77b46c6a5ff8821786121c7"
format_stamp: "Formatted at 2024-05-01 02:08:08 on dist-test-slave-9tkh"
server_key: "1c9fef90c6b313f673d5d1047d79ccaf"
server_key_iv: "42b53d3388fde095a9c518dc1f1e5398"
server_key_version: "encryptionkey@0"
I20240501 02:08:08.832688 28669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:08:08.833273 28684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:08.833441 28669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:08.833513 28669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "34b24cf1f77b46c6a5ff8821786121c7"
format_stamp: "Formatted at 2024-05-01 02:08:08 on dist-test-slave-9tkh"
server_key: "1c9fef90c6b313f673d5d1047d79ccaf"
server_key_iv: "42b53d3388fde095a9c518dc1f1e5398"
server_key_version: "encryptionkey@0"
I20240501 02:08:08.833570 28669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:08.848642 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:08.850257 28669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:08.850380 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:08.853873 28669 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:35589
I20240501 02:08:08.853911 28736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:35589 every 8 connection(s)
I20240501 02:08:08.854205 28669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:08.855015 28737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:08.857034 28737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Bootstrap starting.
I20240501 02:08:08.857527 28737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:08.857739 28737 log.cc:824] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:08.858340 28737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: No bootstrap required, opened a new log
I20240501 02:08:08.859772 28737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } }
I20240501 02:08:08.859896 28737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:08.859913 28737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b24cf1f77b46c6a5ff8821786121c7, State: Initialized, Role: FOLLOWER
I20240501 02:08:08.860018 28737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } }
I20240501 02:08:08.860087 28737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:08.860116 28737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:08.860143 28737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:08.860620 28737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } }
I20240501 02:08:08.860702 28737 leader_election.cc:302] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b24cf1f77b46c6a5ff8821786121c7; no voters:
I20240501 02:08:08.860821 28737 leader_election.cc:288] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:08.861022 28740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:08.861034 28737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:08.861182 28740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 1 LEADER]: Becoming Leader. State: Replica: 34b24cf1f77b46c6a5ff8821786121c7, State: Running, Role: LEADER
I20240501 02:08:08.861292 28740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } }
I20240501 02:08:08.861608 28741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } }
I20240501 02:08:08.861698 28741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:08.861856 28742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b24cf1f77b46c6a5ff8821786121c7. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } }
I20240501 02:08:08.861886 28749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:08.861904 28742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:08.862318 28749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:08.863699 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28669
I20240501 02:08:08.863718 28749 catalog_manager.cc:1347] Generated new cluster ID: 91aaa1bcedb64335b647d0542378190e
I20240501 02:08:08.863759 28749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:08:08.863812 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:08.863979 22577 external_mini_cluster.cc:1428] Setting key 36b5c5baec9939dc59fffb2e5753e685
May 01 02:08:08 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:08.873863 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:08.864584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.22.12.126:35589)
0501 02:08:08.864728 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:08.864738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:08.864764 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:08:08.864904 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:08.864909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:08.864953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:08.865079 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:08.865932 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:08.867075 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:08.867839 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:08.868042 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:08.870550 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:08.870557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:08.870557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:08.870576 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:08.872523 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:08.872849 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:08.872852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:08.872852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:08.872918 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:08.873054 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:08.873055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:08.873055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:08.873302 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:08.873444 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:08.873526 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 02:08:08.873651 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20240501 02:08:08.880440 28749 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:08.880932 28749 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:08.887099 28749 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Generated new TSK 0
I20240501 02:08:08.887306 28749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab.
Entry for principal oryx/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab.
I20240501 02:08:08.917901 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35589
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:08.995388 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:08.995585 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:08.995616 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:08.996924 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:08.996969 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:08.997002 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:08:08.998591 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:08.998798 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:08.998980 28763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.003839 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.004134 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.004593 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.005067 28763 server_base.cc:1034] running on GCE node
I20240501 02:08:09.005251 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.005477 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.006582 28763 hybrid_clock.cc:648] HybridClock initialized: now 1714529289006573 us; error 27 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.009359 28763 init.cc:377] Logged in from keytab as oryx/127.22.12.65@KRBTEST.COM (short username oryx)
I20240501 02:08:09.009814 28763 webserver.cc:460] Webserver started at http://127.22.12.65:36641/ using document root <none> and password file <none>
I20240501 02:08:09.010000 28763 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.010046 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.010145 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:09.011359 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "1c23eec833ec4996a6d8b61316549054"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "68d9f43886092d87e6817dd19e507bb3"
server_key_iv: "77a3a566be84268838ea718bd05cc194"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.011714 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "1c23eec833ec4996a6d8b61316549054"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "68d9f43886092d87e6817dd19e507bb3"
server_key_iv: "77a3a566be84268838ea718bd05cc194"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.012993 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:08:09.013557 28778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.013718 28763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:09.013788 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "1c23eec833ec4996a6d8b61316549054"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "68d9f43886092d87e6817dd19e507bb3"
server_key_iv: "77a3a566be84268838ea718bd05cc194"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.013844 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.030472 28763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.032148 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:09.032249 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.032433 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:09.032694 28763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:09.032723 28763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.032744 28763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:09.032758 28763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.038049 28763 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:45011
I20240501 02:08:09.038118 28891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:45011 every 8 connection(s)
I20240501 02:08:09.038440 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:08:09.042392 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28763
I20240501 02:08:09.042495 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:08:09.042662 22577 external_mini_cluster.cc:1428] Setting key 42f3de12ac2307adccab57fbb47a5199
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:09.046649 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.039136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:47029 (local address 127.22.12.126:35589)
0501 02:08:09.039186 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.039189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.039350 (+ 161us) server_negotiation.cc:407] Connection header received
0501 02:08:09.039819 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.039820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.039833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.039885 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.040893 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.041402 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.041931 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.042076 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.043980 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.043984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.043985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.043992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.045431 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.045847 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.045848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.045848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.045865 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.046042 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.046042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.046043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.046122 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.046164 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.046469 (+ 305us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.046579 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:08:09.046811 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.039113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:47029)
0501 02:08:09.039244 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.039270 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.039720 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.039945 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.039948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.040237 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.040812 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.040817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.041471 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.041472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.041841 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.041842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.041945 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.042459 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.042476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.043895 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.045505 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.045512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.045515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.045769 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.045911 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.045912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.045912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.045990 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.046170 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.046175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.046330 (+ 155us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.046453 (+ 123us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.046619 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20240501 02:08:09.047293 28892 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35589
I20240501 02:08:09.047397 28892 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:09.047600 28892 heartbeater.cc:511] Master 127.22.12.126:35589 requested a full tablet report, sending...
I20240501 02:08:09.048111 28701 ts_manager.cc:194] Registered new tserver with Master: 1c23eec833ec4996a6d8b61316549054 (127.22.12.65:45011)
I20240501 02:08:09.048740 28701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:47029
WARNING: no policy specified for oryx/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab.
Entry for principal oryx/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab.
I20240501 02:08:09.065093 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35589
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:09.141700 28897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:09.141894 28897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:09.141916 28897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:09.143265 28897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:09.143309 28897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:09.143333 28897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:08:09.144946 28897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:09.145152 28897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:09.145339 28897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.149590 28903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.149938 28905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.150372 28902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.150794 28897 server_base.cc:1034] running on GCE node
I20240501 02:08:09.150967 28897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.151168 28897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.152292 28897 hybrid_clock.cc:648] HybridClock initialized: now 1714529289152280 us; error 28 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.154501 28897 init.cc:377] Logged in from keytab as oryx/127.22.12.66@KRBTEST.COM (short username oryx)
I20240501 02:08:09.154938 28897 webserver.cc:460] Webserver started at http://127.22.12.66:41383/ using document root <none> and password file <none>
I20240501 02:08:09.155088 28897 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.155120 28897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.155198 28897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:09.156090 28897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "98108dcd65a24078925d9616fccce6ed"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "f7ab38a5bc13431e08589bac5936ae91"
server_key_iv: "fb2e77f49c3b2dd2bc8877a08f070898"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.156335 28897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "98108dcd65a24078925d9616fccce6ed"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "f7ab38a5bc13431e08589bac5936ae91"
server_key_iv: "fb2e77f49c3b2dd2bc8877a08f070898"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.157353 28897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:08:09.157925 28912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.158066 28897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.158135 28897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "98108dcd65a24078925d9616fccce6ed"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "f7ab38a5bc13431e08589bac5936ae91"
server_key_iv: "fb2e77f49c3b2dd2bc8877a08f070898"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.158190 28897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.171536 28897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.173291 28897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:09.173413 28897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.173609 28897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:09.173902 28897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:09.173933 28897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.173959 28897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:09.173977 28897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.179225 28897 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:46033
I20240501 02:08:09.179262 29025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:46033 every 8 connection(s)
I20240501 02:08:09.179548 28897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:09.187572 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.180277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:56559 (local address 127.22.12.126:35589)
0501 02:08:09.180343 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.180345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.180502 (+ 157us) server_negotiation.cc:407] Connection header received
0501 02:08:09.180999 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.181000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.181009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.181066 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.182159 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.182701 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.183234 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.183354 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.185184 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.185189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.185189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.185196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.186471 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.186838 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.186838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.186839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.186861 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.187065 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.187065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.187065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.187149 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.187193 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.187429 (+ 236us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.187499 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:08:09.187688 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.180232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:56559)
0501 02:08:09.180394 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.180422 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.180882 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.181167 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.181171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.181504 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.182059 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.182063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.182779 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.182780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.183150 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.183151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.183290 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.183776 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.183792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.185100 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.186578 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.186584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.186587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.186778 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.186935 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.186936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.186937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.187007 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.187207 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.187210 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.187326 (+ 116us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.187415 (+ 89us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.187529 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20240501 02:08:09.188176 29026 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35589
I20240501 02:08:09.188275 29026 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:09.188504 29026 heartbeater.cc:511] Master 127.22.12.126:35589 requested a full tablet report, sending...
I20240501 02:08:09.188957 28701 ts_manager.cc:194] Registered new tserver with Master: 98108dcd65a24078925d9616fccce6ed (127.22.12.66:46033)
I20240501 02:08:09.189244 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 28897
I20240501 02:08:09.189340 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:08:09.189527 22577 external_mini_cluster.cc:1428] Setting key dd81128f963969342272b186731c84bb
I20240501 02:08:09.189519 28701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:56559
WARNING: no policy specified for oryx/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab.
Entry for principal oryx/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab.
I20240501 02:08:09.214102 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35589
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:09.291184 29031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:09.291381 29031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:09.291404 29031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:09.292682 29031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:09.292718 29031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:09.292748 29031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:08:09.294267 29031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:09.294437 29031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:09.294623 29031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.298889 29037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.299259 29039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.299695 29036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.300091 29031 server_base.cc:1034] running on GCE node
I20240501 02:08:09.300276 29031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.300498 29031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.301623 29031 hybrid_clock.cc:648] HybridClock initialized: now 1714529289301606 us; error 32 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.304087 29031 init.cc:377] Logged in from keytab as oryx/127.22.12.67@KRBTEST.COM (short username oryx)
I20240501 02:08:09.304514 29031 webserver.cc:460] Webserver started at http://127.22.12.67:46349/ using document root <none> and password file <none>
I20240501 02:08:09.304677 29031 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.304718 29031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.304819 29031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:09.305799 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "878c3f3a84b7499389ce87ce2e75df74"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "1cccbee1655855b532c1697ba236ea91"
server_key_iv: "5740863fa93b4e92bc6b25cbf786d1bd"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.306088 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "878c3f3a84b7499389ce87ce2e75df74"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "1cccbee1655855b532c1697ba236ea91"
server_key_iv: "5740863fa93b4e92bc6b25cbf786d1bd"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.307483 29031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:08:09.308185 29046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.308341 29031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:09.308429 29031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "878c3f3a84b7499389ce87ce2e75df74"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "1cccbee1655855b532c1697ba236ea91"
server_key_iv: "5740863fa93b4e92bc6b25cbf786d1bd"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.308486 29031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.317621 29031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.319638 29031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:09.319751 29031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.319945 29031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:09.320253 29031 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:09.320286 29031 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.320314 29031 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:09.320331 29031 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.325671 29031 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:43727
I20240501 02:08:09.325723 29159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:43727 every 8 connection(s)
I20240501 02:08:09.326001 29031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:08:09.328382 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 29031
I20240501 02:08:09.328481 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:08:09.328667 22577 external_mini_cluster.cc:1428] Setting key 36e694cb4f727f9f18eb4351881cc0bb
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:09.334043 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.326708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:56011 (local address 127.22.12.126:35589)
0501 02:08:09.326774 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.326776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.326905 (+ 129us) server_negotiation.cc:407] Connection header received
0501 02:08:09.327422 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.327424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.327435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.327488 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.328648 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.329297 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.329878 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.329999 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.331804 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.331808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.331808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.331814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.333071 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.333452 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.333452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.333453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.333471 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.333596 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.333597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.333597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.333677 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.333718 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.333910 (+ 192us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.333977 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:08:09.334142 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.326683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:56011)
0501 02:08:09.326798 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.326828 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.327316 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.327547 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.327551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.327844 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.328520 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.328527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.329370 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.329371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.329805 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.329807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.329913 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.330364 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.330376 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.331724 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.333158 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.333163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.333166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.333388 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.333518 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.333518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.333518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.333565 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.333717 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.333719 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.333822 (+ 103us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.333902 (+ 80us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.334002 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20240501 02:08:09.334625 29160 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35589
I20240501 02:08:09.334707 29160 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:09.334894 29160 heartbeater.cc:511] Master 127.22.12.126:35589 requested a full tablet report, sending...
I20240501 02:08:09.335258 28701 ts_manager.cc:194] Registered new tserver with Master: 878c3f3a84b7499389ce87ce2e75df74 (127.22.12.67:43727)
I20240501 02:08:09.335623 28701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:56011
I20240501 02:08:09.340304 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.125.keytab.
Entry for principal oryx/127.22.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.125.keytab.
I20240501 02:08:09.363502 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:39095
--webserver_interface=127.22.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.125.keytab
--principal=oryx/127.22.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35589,127.22.12.125:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:09.454072 29165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:09.454329 29165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:09.454360 29165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:09.456497 29165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:09.456553 29165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:09.456573 29165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:09.456598 29165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:09.456615 29165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:09.459035 29165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35589,127.22.12.125:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.125.keytab
--principal=oryx/127.22.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:39095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
W20240501 02:08:09.459170 29165 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.12.126:35589,127.22.12.125:39095'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20240501 02:08:09.459311 29165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:09.459517 29165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.464413 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.464830 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.465232 29165 server_base.cc:1034] running on GCE node
W20240501 02:08:09.465611 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.465931 29165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.466212 29165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.467347 29165 hybrid_clock.cc:648] HybridClock initialized: now 1714529289467329 us; error 33 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.470310 29165 init.cc:377] Logged in from keytab as oryx/127.22.12.125@KRBTEST.COM (short username oryx)
I20240501 02:08:09.470742 29165 webserver.cc:460] Webserver started at http://127.22.12.125:42309/ using document root <none> and password file <none>
I20240501 02:08:09.470957 29165 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.470999 29165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.471097 29165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:09.472071 29165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/instance:
uuid: "2cab249f87cf4277a357ab25503066b9"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "0d5cafbdc7690c735f1926582900f18d"
server_key_iv: "ce41d52c1a73ff719ea4c5d692ed74e1"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.472349 29165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal/instance:
uuid: "2cab249f87cf4277a357ab25503066b9"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "0d5cafbdc7690c735f1926582900f18d"
server_key_iv: "ce41d52c1a73ff719ea4c5d692ed74e1"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.473531 29165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:08:09.474184 29180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.474355 29165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:09.474449 29165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
uuid: "2cab249f87cf4277a357ab25503066b9"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "0d5cafbdc7690c735f1926582900f18d"
server_key_iv: "ce41d52c1a73ff719ea4c5d692ed74e1"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.474511 29165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.489931 29165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.492061 29165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:09.492183 29165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.495860 29165 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.125:39095
I20240501 02:08:09.495882 29232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.125:39095 every 8 connection(s)
I20240501 02:08:09.496284 29165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
I20240501 02:08:09.497267 29233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:09.499316 29233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:08:09.501399 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 29165
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.125@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
I20240501 02:08:09.509522 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.500253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.22.12.126:35589)
0501 02:08:09.500328 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.500330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.500534 (+ 204us) server_negotiation.cc:407] Connection header received
0501 02:08:09.501170 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.501172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.501189 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.501250 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.502324 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.503086 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.503640 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.503807 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.506592 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.506597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.506597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.506604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.508226 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.508696 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.508697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.508698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.508724 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.508905 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.508906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.508906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.509018 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.509063 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.509346 (+ 283us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.509429 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 02:08:09.509585 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.500203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:36800)
0501 02:08:09.500381 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.500417 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.501054 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.501314 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.501319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.501747 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.502182 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.502188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.503158 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.503159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.503569 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.503570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.503699 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.504798 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.504822 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.506485 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.508311 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.508316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.508319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.508620 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.508787 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.508787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.508788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.508853 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.509070 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.509073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.509213 (+ 140us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.509323 (+ 110us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.509426 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20240501 02:08:09.510154 29233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:08:09.510547 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.501812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.22.12.125:39095)
0501 02:08:09.501949 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.501956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.502017 (+ 61us) server_negotiation.cc:407] Connection header received
0501 02:08:09.502086 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.502089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.502105 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.502169 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.502993 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.504099 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.504634 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.504837 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.507530 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.507538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.507538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.507543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.509198 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.509549 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.509551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.509551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.509577 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.509872 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.509873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.509873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.509990 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.510113 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.510202 (+ 89us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.510435 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.125@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
I20240501 02:08:09.515352 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.510392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.0.0.1:51672)
0501 02:08:09.510424 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.510434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.510467 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.510730 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.510731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.510839 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.510983 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.510984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.511581 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.511582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.511918 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.511920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.511969 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.512345 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.512350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.513392 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.514768 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.514769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.514769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.514884 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.514988 (+ 104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.514989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.514989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.515008 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.515142 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.515142 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.515172 (+ 30us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.515211 (+ 39us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.515285 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20240501 02:08:09.515424 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.510483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.22.12.125:39095)
0501 02:08:09.510616 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.510618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.510624 (+ 6us) server_negotiation.cc:407] Connection header received
0501 02:08:09.510646 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.510646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.510654 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.510697 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.511044 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.511512 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.511971 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.512088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.513462 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.513465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.513466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.513488 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.514712 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.514934 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.514935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.514935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.514951 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.515037 (+ 86us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.515037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.515037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.515108 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.515147 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.515231 (+ 84us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.515339 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":121,"thread_start_us":105,"threads_started":1}
I20240501 02:08:09.516832 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Bootstrap starting.
I20240501 02:08:09.517333 29233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:09.517567 29233 log.cc:824] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:09.518301 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: No bootstrap required, opened a new log
I20240501 02:08:09.519584 29233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } }
I20240501 02:08:09.519698 29233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:09.519716 29233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cab249f87cf4277a357ab25503066b9, State: Initialized, Role: FOLLOWER
I20240501 02:08:09.519793 29233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } }
I20240501 02:08:09.520018 29233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:09.519999 29240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } } }
I20240501 02:08:09.520087 29240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: This master's current role is: FOLLOWER
W20240501 02:08:09.521580 29251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:08:09.521628 29251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:08:09.523115 29165 master_runner.cc:428] Detected that this master 2cab249f87cf4277a357ab25503066b9 is joining an existing cluster
I20240501 02:08:09.523161 29165 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.12.125:39095
I20240501 02:08:09.529489 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.524395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.22.12.125:39095)
0501 02:08:09.524428 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.524430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.524481 (+ 51us) server_negotiation.cc:407] Connection header received
0501 02:08:09.524536 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.524536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.524543 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.524593 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.525042 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.525522 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.526225 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.526371 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.527260 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.527262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.527263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.527268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.528601 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.528918 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.528919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.528919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.528934 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.529066 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.529067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.529067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.529157 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.529200 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.529343 (+ 143us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.529420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:08:09.529613 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.524238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.0.0.1:51678)
0501 02:08:09.524398 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.524409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.524484 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.524605 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.524606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.524779 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.524976 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.524978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.525592 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.525593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.526137 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.526139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.526223 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.526826 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.526830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.527176 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.528672 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.528673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.528673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.528854 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.528980 (+ 126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.528981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.528981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.529011 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.529202 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.529203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.529250 (+ 47us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.529328 (+ 78us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.529503 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":69,"thread_start_us":104,"threads_started":1}
I20240501 02:08:09.545843 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.524422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.22.12.126:35589)
0501 02:08:09.524461 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.524463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.524628 (+ 165us) server_negotiation.cc:407] Connection header received
0501 02:08:09.524714 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.524715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.524725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.524753 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.525335 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.525880 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.526413 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.526586 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.543567 (+ 16981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.543574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.543574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.543582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.544836 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.545133 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.545135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.545135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.545158 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.545299 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.545300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.545301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.545405 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.545460 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.545647 (+ 187us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.545751 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:08:09.545855 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.524251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:36808)
0501 02:08:09.524572 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.524579 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.524650 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.524816 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.524817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.525008 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.525252 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.525254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.525944 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.525945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.526325 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.526326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.526634 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.543161 (+ 16527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.543178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.543449 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.544930 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.544931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.544931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.545071 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.545207 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.545208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.545208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.545251 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.545459 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.545460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.545528 (+ 68us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.545615 (+ 87us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.545733 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"spinlock_wait_cycles":6912,"thread_start_us":93,"threads_started":1}
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.126@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
I20240501 02:08:09.554376 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.548048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.0.0.1:51692)
0501 02:08:09.548193 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.548214 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.548287 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.548436 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.548439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.548743 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.548914 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.548916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.549469 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.549470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.549821 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.549822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.549880 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.550427 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.550432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.551979 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.553541 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.553547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.553549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.553733 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.553874 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.553874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.553874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.553899 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.554096 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.554101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.554145 (+ 44us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.554213 (+ 68us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.554295 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20240501 02:08:09.554425 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.548093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.22.12.125:39095)
0501 02:08:09.548137 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.548139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.548273 (+ 134us) server_negotiation.cc:407] Connection header received
0501 02:08:09.548368 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.548369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.548378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.548425 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.548989 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.549402 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.549891 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.550042 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.552067 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.552071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.552071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.552078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.553458 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.553795 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.553796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.553796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.553817 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.553940 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.553941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.553941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.554045 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.554105 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.554248 (+ 143us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.554328 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:08:09.555020 28701 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.22.12.125:39095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2cab249f87cf4277a357ab25503066b9"
member_type: NON_VOTER
last_known_addr {
host: "127.22.12.125"
port: 39095
}
attrs {
promote: true
}
}
dest_uuid: "34b24cf1f77b46c6a5ff8821786121c7"
cas_config_opid_index: -1
I20240501 02:08:09.555284 28701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } }
I20240501 02:08:09.555724 29260 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2cab249f87cf4277a357ab25503066b9 (127.22.12.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } }
I20240501 02:08:09.556036 29207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:09.556586 29260 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.22.12.125:39095
I20240501 02:08:09.556670 29260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } }
I20240501 02:08:09.556735 29260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:09.556824 29165 master.cc:537] Master@127.22.12.125:39095 shutting down...
I20240501 02:08:09.556927 29207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Refusing update from remote peer 34b24cf1f77b46c6a5ff8821786121c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 02:08:09.557529 29261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 02:08:09.558378 28685 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20240501 02:08:09.559175 28685 consensus_peers.cc:485] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 -> Peer 2cab249f87cf4277a357ab25503066b9 (127.22.12.125:39095): Couldn't send request to peer 2cab249f87cf4277a357ab25503066b9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240501 02:08:09.559229 29165 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:09.559394 29165 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:09.559448 29165 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: stopping tablet replica
I20240501 02:08:09.571664 29165 master.cc:552] Master@127.22.12.125:39095 shutdown complete.
I20240501 02:08:09.571717 29165 master_runner.cc:305] Clearing existing system tablet
I20240501 02:08:09.571831 29165 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.571871 29165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.572453 29165 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:09.573029 29265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.573168 29165 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.573211 29165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:08:09.573230 29165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
uuid: "2cab249f87cf4277a357ab25503066b9"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "0d5cafbdc7690c735f1926582900f18d"
server_key_iv: "ce41d52c1a73ff719ea4c5d692ed74e1"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.573817 29165 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:09.575178 29165 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240501 02:08:09.575234 29165 log.cc:1196] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20240501 02:08:09.575385 29165 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Deleting consensus metadata
I20240501 02:08:09.575484 29165 master_runner.cc:315] Copying system tablet from 127.22.12.126:35589
I20240501 02:08:09.576181 29165 tablet_copy_client.cc:308] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: tablet copy: Beginning tablet copy session from remote peer at address 127.22.12.126:35589
I20240501 02:08:09.581395 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.576508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:36824)
0501 02:08:09.576626 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.576636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.576720 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.576849 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.576850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.577021 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.577189 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.577191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.577812 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.577813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.578201 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.578202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.578264 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.578892 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.578897 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.579142 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.580581 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.580582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.580582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.580720 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.580850 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.580851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.580851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.580887 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.581072 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.581073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.581119 (+ 46us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.581176 (+ 57us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.581282 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20240501 02:08:09.581389 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.576559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.22.12.126:35589)
0501 02:08:09.576602 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.576604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.576702 (+ 98us) server_negotiation.cc:407] Connection header received
0501 02:08:09.576781 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.576782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.576793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.576841 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.577266 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.577744 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.578274 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.578450 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.579222 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.579227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.579228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.579234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.580508 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.580778 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.580779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.580779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.580802 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.580934 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.580934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.580935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.581024 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.581078 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.581209 (+ 131us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.581282 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:08:09.581861 28721 tablet_copy_service.cc:140] P 34b24cf1f77b46c6a5ff8821786121c7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2cab249f87cf4277a357ab25503066b9 ({username='oryx', principal='oryx/127.22.12.125@KRBTEST.COM'} at 127.0.0.1:36824)
I20240501 02:08:09.581945 28721 tablet_copy_service.cc:161] P 34b24cf1f77b46c6a5ff8821786121c7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2cab249f87cf4277a357ab25503066b9 at {username='oryx', principal='oryx/127.22.12.125@KRBTEST.COM'} at 127.0.0.1:36824: session id = 2cab249f87cf4277a357ab25503066b9-00000000000000000000000000000000
I20240501 02:08:09.582473 28721 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 02:08:09.583039 29165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240501 02:08:09.584023 29165 tablet_copy_client.cc:783] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: tablet copy: Starting download of 0 data blocks...
I20240501 02:08:09.584252 29165 tablet_copy_client.cc:647] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: tablet copy: Starting download of 1 WAL segments...
I20240501 02:08:09.586165 29165 tablet_copy_client.cc:523] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 02:08:09.588101 28721 tablet_copy_service.cc:342] P 34b24cf1f77b46c6a5ff8821786121c7: Request end of tablet copy session 2cab249f87cf4277a357ab25503066b9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.12.125@KRBTEST.COM'} at 127.0.0.1:36824
I20240501 02:08:09.588160 28721 tablet_copy_service.cc:433] P 34b24cf1f77b46c6a5ff8821786121c7: ending tablet copy session 2cab249f87cf4277a357ab25503066b9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2cab249f87cf4277a357ab25503066b9
I20240501 02:08:09.590716 29165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.592365 29276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.592814 29275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.593220 29278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.593662 29165 server_base.cc:1034] running on GCE node
I20240501 02:08:09.593751 29165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.593885 29165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.595000 29165 hybrid_clock.cc:648] HybridClock initialized: now 1714529289594988 us; error 30 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.596448 29165 init.cc:377] Logged in from keytab as oryx/127.22.12.125@KRBTEST.COM (short username oryx)
I20240501 02:08:09.596706 29165 webserver.cc:460] Webserver started at http://127.22.12.125:34075/ using document root <none> and password file <none>
I20240501 02:08:09.596812 29165 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.596855 29165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.597507 29165 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:09.597995 29285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.598107 29165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.598158 29165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
uuid: "2cab249f87cf4277a357ab25503066b9"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "0d5cafbdc7690c735f1926582900f18d"
server_key_iv: "ce41d52c1a73ff719ea4c5d692ed74e1"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.598215 29165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.623026 29165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.624692 29165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.627673 29165 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.125:39095
I20240501 02:08:09.627825 29337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.125:39095 every 8 connection(s)
I20240501 02:08:09.628038 29165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
I20240501 02:08:09.628698 29338 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:08:09.629050 29338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Bootstrap starting.
I20240501 02:08:09.631182 29338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:08:09.631356 29338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: Bootstrap complete.
I20240501 02:08:09.631542 29338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } }
I20240501 02:08:09.631609 29338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2cab249f87cf4277a357ab25503066b9, State: Initialized, Role: LEARNER
I20240501 02:08:09.631642 29338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } }
I20240501 02:08:09.631819 29338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:09.631804 29341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } }
I20240501 02:08:09.631891 29341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: This master's current role is: LEARNER
I20240501 02:08:09.633148 29352 catalog_manager.cc:1259] Loaded cluster ID: 91aaa1bcedb64335b647d0542378190e
I20240501 02:08:09.633220 29352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: loading cluster ID for follower catalog manager: success
I20240501 02:08:09.634167 29352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: acquiring CA information for follower catalog manager: success
I20240501 02:08:09.634428 29352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:08:09.668424 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.662403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.22.12.125:39095)
0501 02:08:09.662520 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.662522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.662578 (+ 56us) server_negotiation.cc:407] Connection header received
0501 02:08:09.662617 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.662618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.662629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.662688 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.663332 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.664045 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.664577 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.664748 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.665852 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.665857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.665857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.665863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.667485 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.667787 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.667788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.667788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.667809 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.667933 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.667933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.667934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.668029 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.668131 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.668216 (+ 85us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.668300 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.22.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.124.keytab.
Entry for principal oryx/127.22.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.124.keytab.
I20240501 02:08:09.693266 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:38171
--webserver_interface=127.22.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.124.keytab
--principal=oryx/127.22.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:09.771063 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:09.771258 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:09.771278 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:09.772552 29356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:09.772593 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:09.772604 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:09.772615 29356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:09.772625 29356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:09.774106 29356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.124.keytab
--principal=oryx/127.22.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:09.774289 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:09.774448 29356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.778755 29362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.779040 29361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.779515 29364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.779963 29356 server_base.cc:1034] running on GCE node
I20240501 02:08:09.780206 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.780561 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.781706 29356 hybrid_clock.cc:648] HybridClock initialized: now 1714529289781690 us; error 30 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.784181 29356 init.cc:377] Logged in from keytab as oryx/127.22.12.124@KRBTEST.COM (short username oryx)
I20240501 02:08:09.784585 29356 webserver.cc:460] Webserver started at http://127.22.12.124:35351/ using document root <none> and password file <none>
I20240501 02:08:09.784744 29356 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.784785 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.784893 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:09.785859 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/instance:
uuid: "7a6f9aecf5a24a97b897b31ef3f0058c"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "8b5c480d0906d755d43c0ee4e972fca9"
server_key_iv: "70f7753f563343e8c345c6fbe1add53a"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.786137 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal/instance:
uuid: "7a6f9aecf5a24a97b897b31ef3f0058c"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "8b5c480d0906d755d43c0ee4e972fca9"
server_key_iv: "70f7753f563343e8c345c6fbe1add53a"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.787341 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:08:09.787992 29371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.788142 29356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:09.788214 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
uuid: "7a6f9aecf5a24a97b897b31ef3f0058c"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "8b5c480d0906d755d43c0ee4e972fca9"
server_key_iv: "70f7753f563343e8c345c6fbe1add53a"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.788271 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.838004 29356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.839777 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:09.839892 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.843525 29356 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.124:38171
I20240501 02:08:09.843564 29423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.124:38171 every 8 connection(s)
I20240501 02:08:09.843850 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
I20240501 02:08:09.844630 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:09.846014 29424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:08:09.847985 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 29356
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.124@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.12.124@KRBTEST.COM
I20240501 02:08:09.854166 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.846672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:36836)
0501 02:08:09.846823 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.846852 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.847268 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.847544 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.847548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.847845 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.848122 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.848127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.848949 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.848950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.849533 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.849535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.849626 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.850111 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.850125 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.851535 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.852991 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.852996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.852998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.853179 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.853321 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.853322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.853323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.853391 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.853643 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.853647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.853761 (+ 114us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.853910 (+ 149us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.854022 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20240501 02:08:09.854562 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.846789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.22.12.126:35589)
0501 02:08:09.846848 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.846850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.846913 (+ 63us) server_negotiation.cc:407] Connection header received
0501 02:08:09.847371 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.847373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.847383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.847440 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.848221 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.848877 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.849599 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.849757 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.851622 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.851626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.851626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.851633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.852922 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.853247 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.853248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.853248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.853272 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.853464 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.853465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.853465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.853578 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.853650 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.854379 (+ 729us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.854451 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:08:09.854938 29424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:08:09.856184 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.848488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.22.12.124:38171)
0501 02:08:09.848614 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.848620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.848631 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:08:09.848646 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.848647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.848660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.848719 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.849500 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.850225 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.850913 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.851087 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.853267 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.853273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.853273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.853277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.855175 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.855508 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.855510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.855510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.855533 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.855669 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.855669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.855670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.855762 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.855903 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.855999 (+ 96us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.856071 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.124@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
I20240501 02:08:09.860047 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.855160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.0.0.1:51728)
0501 02:08:09.855196 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.855204 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.855232 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.855412 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.855413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.855552 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.855715 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.855717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.856500 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.856501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.856830 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.856831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.856891 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.857263 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.857266 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.858220 (+ 954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.859439 (+ 1219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.859440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.859440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.859555 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.859695 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.859696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.859696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.859715 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.859852 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.859852 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.859883 (+ 31us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.859925 (+ 42us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.859984 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20240501 02:08:09.860067 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.855234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.22.12.125:39095)
0501 02:08:09.855296 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.855298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.855305 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:08:09.855326 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.855327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.855342 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.855394 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.855797 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.856444 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.856895 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.857051 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.858290 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.858293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.858294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.858299 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.859372 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.859621 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.859622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.859622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.859645 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.859745 (+ 100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.859745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.859745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.859817 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.859856 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.859941 (+ 85us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.860008 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:08:09.860399 29424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 38171 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.124@KRBTEST.COM for oryx/127.22.12.124@KRBTEST.COM
I20240501 02:08:09.865110 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.860617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.0.0.1:36660)
0501 02:08:09.860656 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.860664 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.860697 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.860777 (+ 80us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.860777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.860906 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.861030 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.861031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.861613 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.861614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.861958 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.861959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.862017 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.862390 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.862393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.863403 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.864527 (+ 1124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.864528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.864528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.864644 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.864750 (+ 106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.864750 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.864751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.864769 (+ 18us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.864897 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.864897 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.864926 (+ 29us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.864968 (+ 42us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.865032 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 02:08:09.865159 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.860656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.22.12.124:38171)
0501 02:08:09.860688 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.860689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.860696 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:08:09.860730 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.860731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.860739 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.860774 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.861078 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.861552 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.862127 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.862309 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.863468 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.863470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.863470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.863475 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.864465 (+ 990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.864694 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.864695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.864695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.864709 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.864799 (+ 90us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.864799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.864800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.864861 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.864899 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.864982 (+ 83us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.865071 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":17}
I20240501 02:08:09.866506 29424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Bootstrap starting.
I20240501 02:08:09.867055 29424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:09.867272 29424 log.cc:824] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:09.867910 29424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: No bootstrap required, opened a new log
I20240501 02:08:09.869112 29424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 38171 } }
I20240501 02:08:09.869225 29424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:09.869246 29424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f9aecf5a24a97b897b31ef3f0058c, State: Initialized, Role: FOLLOWER
I20240501 02:08:09.869333 29424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 38171 } }
I20240501 02:08:09.869578 29424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:09.869556 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 38171 } } }
I20240501 02:08:09.869627 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [sys.catalog]: This master's current role is: FOLLOWER
W20240501 02:08:09.871006 29440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:08:09.871058 29440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:08:09.872972 29356 master_runner.cc:428] Detected that this master 7a6f9aecf5a24a97b897b31ef3f0058c is joining an existing cluster
I20240501 02:08:09.873016 29356 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.12.124:38171
I20240501 02:08:09.875260 29429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:09.875346 29429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 38171 } }
I20240501 02:08:09.875579 29429 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589), 2cab249f87cf4277a357ab25503066b9 (127.22.12.125:39095)
I20240501 02:08:09.875730 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b24cf1f77b46c6a5ff8821786121c7" is_pre_election: true
I20240501 02:08:09.875955 29312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cab249f87cf4277a357ab25503066b9" is_pre_election: true
I20240501 02:08:09.876034 29312 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Handling vote request from an unknown peer 7a6f9aecf5a24a97b897b31ef3f0058c
I20240501 02:08:09.876067 29312 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 7a6f9aecf5a24a97b897b31ef3f0058c for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 02:08:09.876292 29373 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a6f9aecf5a24a97b897b31ef3f0058c; no voters: 2cab249f87cf4277a357ab25503066b9, 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:09.876410 29429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:09.895602 29429 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 02:08:09.901093 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.873898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.0.0.1:51730)
0501 02:08:09.874129 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.874138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.874213 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.874411 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.874412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.894917 (+ 20505us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.895190 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.895195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896648 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.896649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.897151 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.897153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.897211 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.898081 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.898084 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.898432 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.900116 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.900117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.900298 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.900454 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.900455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.900489 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.900733 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.900734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.900783 (+ 49us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.900840 (+ 57us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.900974 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":63,"threads_started":1}
I20240501 02:08:09.901190 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.873941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.22.12.126:35589)
0501 02:08:09.873985 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.873987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.874358 (+ 371us) server_negotiation.cc:407] Connection header received
0501 02:08:09.874446 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.874448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.874458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.874485 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.895350 (+ 20865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896044 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.896780 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896947 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.898016 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.898021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.898021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.898029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.899875 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900246 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.900247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.900247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.900273 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900719 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.900720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.900720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.900820 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.900857 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.901034 (+ 177us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.901110 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:08:09.901511 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.874018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:36842)
0501 02:08:09.874306 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.874313 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.874380 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.874587 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.874587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.894915 (+ 20328us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.895206 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.895208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896140 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.896141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.896696 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.896697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.897001 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.897558 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.897561 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.897921 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.899959 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.899960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.899960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.900159 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.900612 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.900613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.900641 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.900897 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.900897 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.900930 (+ 33us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.901284 (+ 354us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.901428 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20240501 02:08:09.901979 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.874054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.22.12.125:39095)
0501 02:08:09.874087 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.874089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.874193 (+ 104us) server_negotiation.cc:407] Connection header received
0501 02:08:09.874298 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.874299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.874310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.874356 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.895865 (+ 21509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896558 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.897319 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.897465 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.898512 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.898516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.898516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.898523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.900033 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900360 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.900360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.900361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.900383 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.900540 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.900541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.900541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.900651 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.900708 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.901800 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.901899 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:08:09.902114 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.874044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.22.12.124:38171)
0501 02:08:09.874077 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.874079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.874211 (+ 132us) server_negotiation.cc:407] Connection header received
0501 02:08:09.874257 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.874258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.874268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.874310 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.895794 (+ 21484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896451 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.897142 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.897271 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.898873 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.898876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.898876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.898881 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.901186 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.901531 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.901531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.901532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.901548 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.901673 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.901674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.901674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.901761 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.901793 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.901947 (+ 154us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.902021 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:08:09.902379 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.873903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.0.0.1:36664)
0501 02:08:09.874131 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.874138 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.874203 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.874352 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.874361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.895380 (+ 21019us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.895706 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.895708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.896547 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.896548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.897063 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.897064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.897328 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.898323 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.898327 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.898790 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.901260 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.901261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.901261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.901465 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.901600 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.901600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.901600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.901623 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.901824 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.901825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.901852 (+ 27us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.902210 (+ 358us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.902287 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"spinlock_wait_cycles":6912,"thread_start_us":84,"threads_started":1}
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529288, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.126@KRBTEST.COM for oryx/127.22.12.124@KRBTEST.COM
I20240501 02:08:09.907598 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28763
I20240501 02:08:09.909507 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.903624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.0.0.1:36678)
0501 02:08:09.903675 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.903686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.903729 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.903956 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.903957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.904122 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.904310 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.904312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.904818 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.904819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.905362 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.905364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.905453 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.905809 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.905813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.907008 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.908760 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.908761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.908762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.908904 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.909027 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.909028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.909028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.909055 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.909219 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.909219 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.909260 (+ 41us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.909317 (+ 57us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.909413 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20240501 02:08:09.909505 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.903797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.22.12.124:38171)
0501 02:08:09.903845 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.903847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.903853 (+ 6us) server_negotiation.cc:407] Connection header received
0501 02:08:09.903885 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.903886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.903894 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.903933 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.904375 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.904755 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.905593 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.905763 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.907094 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.907097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.907097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.907103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.908687 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.908960 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.908961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.908961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.908980 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.909101 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.909101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.909102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.909174 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.909222 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.909344 (+ 122us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.909414 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:08:09.915030 28701 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.22.12.124:38171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c"
member_type: NON_VOTER
last_known_addr {
host: "127.22.12.124"
port: 38171
}
attrs {
promote: true
}
}
dest_uuid: "34b24cf1f77b46c6a5ff8821786121c7"
cas_config_opid_index: 5
I20240501 02:08:09.915205 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:45011
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:35589
--webserver_port=36641
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:08:09.915273 28701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:09.915733 29260 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7a6f9aecf5a24a97b897b31ef3f0058c (127.22.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.916431 29260 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.22.12.124:38171
I20240501 02:08:09.916505 29260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.916559 29260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:09.917323 29398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 1 FOLLOWER]: Refusing update from remote peer 34b24cf1f77b46c6a5ff8821786121c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 02:08:09.917603 29356 master.cc:537] Master@127.22.12.124:38171 shutting down...
I20240501 02:08:09.917572 29449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 02:08:09.918151 28687 consensus_peers.cc:485] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 -> Peer 7a6f9aecf5a24a97b897b31ef3f0058c (127.22.12.124:38171): Couldn't send request to peer 7a6f9aecf5a24a97b897b31ef3f0058c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240501 02:08:09.919133 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.916717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.0.0.1:51746)
0501 02:08:09.916804 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.916814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.916847 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.917128 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.917129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.917131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:09.917260 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.917261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.918088 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.918089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.918878 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.918887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.918950 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.918996 (+ 46us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.919026 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.919072 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20240501 02:08:09.919956 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.916929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.22.12.125:39095)
0501 02:08:09.916971 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.916972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.916979 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:08:09.917002 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.917003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.917031 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.917099 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:09.917337 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.918009 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.919331 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.919742 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.919828 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.919850 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.919892 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:08:09.920146 29312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Refusing update from remote peer 34b24cf1f77b46c6a5ff8821786121c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 02:08:09.920526 29449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 02:08:09.920723 29356 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:09.920796 29356 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:09.920831 29356 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: stopping tablet replica
I20240501 02:08:09.921059 29312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7a6f9aecf5a24a97b897b31ef3f0058c (127.22.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.922119 29341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b24cf1f77b46c6a5ff8821786121c7. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.922191 29341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: This master's current role is: LEARNER
I20240501 02:08:09.922226 29341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: NON_VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: true } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.922260 29341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: This master's current role is: LEARNER
I20240501 02:08:09.922863 29260 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: attempting to promote NON_VOTER 2cab249f87cf4277a357ab25503066b9 to VOTER
I20240501 02:08:09.923017 29260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:09.923460 29312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 LEARNER]: Refusing update from remote peer 34b24cf1f77b46c6a5ff8821786121c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 02:08:09.923511 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.923412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.0.0.1:36694)
0501 02:08:09.923447 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.923486 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20240501 02:08:09.923709 29453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 02:08:09.923770 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.923697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.0.0.1:36704)
0501 02:08:09.923733 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.923752 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240501 02:08:09.923939 28687 consensus_peers.cc:485] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 -> Peer 7a6f9aecf5a24a97b897b31ef3f0058c (127.22.12.124:38171): Couldn't send request to peer 7a6f9aecf5a24a97b897b31ef3f0058c. Status: Network error: Client connection negotiation failed: client connection to 127.22.12.124:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 02:08:09.924296 29449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2cab249f87cf4277a357ab25503066b9 (127.22.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.924453 29312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2cab249f87cf4277a357ab25503066b9 (127.22.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.925369 29261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.925453 29261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:09.925424 29452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.925487 29452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:08:09.925599 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34b24cf1f77b46c6a5ff8821786121c7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.925658 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:09.933530 29356 master.cc:552] Master@127.22.12.124:38171 shutdown complete.
I20240501 02:08:09.933573 29356 master_runner.cc:305] Clearing existing system tablet
I20240501 02:08:09.933702 29356 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.933739 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.934288 29356 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.934717 29456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.934809 29356 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.934851 29356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.934875 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
uuid: "7a6f9aecf5a24a97b897b31ef3f0058c"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "8b5c480d0906d755d43c0ee4e972fca9"
server_key_iv: "70f7753f563343e8c345c6fbe1add53a"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.935456 29356 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:09.936710 29356 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240501 02:08:09.936766 29356 log.cc:1196] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20240501 02:08:09.936930 29356 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Deleting consensus metadata
I20240501 02:08:09.937034 29356 master_runner.cc:315] Copying system tablet from 127.22.12.126:35589
I20240501 02:08:09.937701 29356 tablet_copy_client.cc:308] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.12.126:35589
I20240501 02:08:09.943559 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.938855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.22.12.126:35589)
0501 02:08:09.938901 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:09.938903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:09.939013 (+ 110us) server_negotiation.cc:407] Connection header received
0501 02:08:09.939072 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:09.939073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:09.939084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:09.939128 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:09.939542 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.940014 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.940530 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.940688 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.941385 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:09.941389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:09.941390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:09.941397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:09.942684 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.942958 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.942959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.942960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.942983 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:09.943106 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:09.943107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:09.943107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:09.943212 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:09.943264 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:09.943398 (+ 134us) server_negotiation.cc:299] Negotiation successful
0501 02:08:09.943466 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:08:09.943571 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:09.938803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:36858)
0501 02:08:09.938943 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:09.938951 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:09.939011 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:09.939139 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:09.939140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:09.939298 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:09.939467 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.939468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.940081 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:09.940082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:09.940461 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:09.940463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:09.940522 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:09.941068 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:09.941071 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:09.941308 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:09.942761 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.942762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.942762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.942895 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.943037 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:09.943037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:09.943038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:09.943061 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:09.943260 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:09.943260 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:09.943303 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:08:09.943356 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 02:08:09.943460 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20240501 02:08:09.943899 28721 tablet_copy_service.cc:140] P 34b24cf1f77b46c6a5ff8821786121c7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7a6f9aecf5a24a97b897b31ef3f0058c ({username='oryx', principal='oryx/127.22.12.124@KRBTEST.COM'} at 127.0.0.1:36858)
I20240501 02:08:09.943953 28721 tablet_copy_service.cc:161] P 34b24cf1f77b46c6a5ff8821786121c7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7a6f9aecf5a24a97b897b31ef3f0058c at {username='oryx', principal='oryx/127.22.12.124@KRBTEST.COM'} at 127.0.0.1:36858: session id = 7a6f9aecf5a24a97b897b31ef3f0058c-00000000000000000000000000000000
I20240501 02:08:09.944375 28721 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 02:08:09.944787 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240501 02:08:09.945650 29356 tablet_copy_client.cc:783] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: tablet copy: Starting download of 0 data blocks...
I20240501 02:08:09.945767 29356 tablet_copy_client.cc:647] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: tablet copy: Starting download of 1 WAL segments...
I20240501 02:08:09.946463 29356 tablet_copy_client.cc:523] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 02:08:09.947227 28721 tablet_copy_service.cc:342] P 34b24cf1f77b46c6a5ff8821786121c7: Request end of tablet copy session 7a6f9aecf5a24a97b897b31ef3f0058c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.12.124@KRBTEST.COM'} at 127.0.0.1:36858
I20240501 02:08:09.947280 28721 tablet_copy_service.cc:433] P 34b24cf1f77b46c6a5ff8821786121c7: ending tablet copy session 7a6f9aecf5a24a97b897b31ef3f0058c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7a6f9aecf5a24a97b897b31ef3f0058c
I20240501 02:08:09.949446 29356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:09.950987 29466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.951853 29467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:09.952178 29469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:09.952610 29356 server_base.cc:1034] running on GCE node
I20240501 02:08:09.952718 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:09.952957 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:09.954072 29356 hybrid_clock.cc:648] HybridClock initialized: now 1714529289954067 us; error 25 us; skew 500 ppm
May 01 02:08:09 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529289, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:09.955308 29356 init.cc:377] Logged in from keytab as oryx/127.22.12.124@KRBTEST.COM (short username oryx)
I20240501 02:08:09.955543 29356 webserver.cc:460] Webserver started at http://127.22.12.124:45943/ using document root <none> and password file <none>
I20240501 02:08:09.955648 29356 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:09.955679 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:09.956214 29356 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:09.956645 29476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.956737 29356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:09.956771 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
uuid: "7a6f9aecf5a24a97b897b31ef3f0058c"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "8b5c480d0906d755d43c0ee4e972fca9"
server_key_iv: "70f7753f563343e8c345c6fbe1add53a"
server_key_version: "encryptionkey@0"
I20240501 02:08:09.956825 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:09.987115 29356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:09.988629 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:09.991266 29356 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.124:38171
I20240501 02:08:09.991319 29528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.124:38171 every 8 connection(s)
I20240501 02:08:09.991588 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
I20240501 02:08:09.992225 29529 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:08:09.992578 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Bootstrap starting.
I20240501 02:08:09.994506 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:08:09.994733 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: Bootstrap complete.
I20240501 02:08:09.994913 29529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:09.994979 29529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a6f9aecf5a24a97b897b31ef3f0058c, State: Initialized, Role: LEARNER
I20240501 02:08:09.995012 29529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:09.995185 29529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:09.995184 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } } }
I20240501 02:08:09.995447 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c [sys.catalog]: This master's current role is: LEARNER
I20240501 02:08:09.996739 29543 catalog_manager.cc:1259] Loaded cluster ID: 91aaa1bcedb64335b647d0542378190e
I20240501 02:08:09.996814 29543 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: loading cluster ID for follower catalog manager: success
I20240501 02:08:09.997850 29543 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: acquiring CA information for follower catalog manager: success
I20240501 02:08:09.998107 29543 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a6f9aecf5a24a97b897b31ef3f0058c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240501 02:08:10.032178 29450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:10.032433 29450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:10.032469 29450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:10.034451 29450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:10.034562 29450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:10.034621 29450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:08:10.037078 29450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:45011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=36641
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:10.037328 29450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:10.037556 29450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:10.049997 29549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:10.050431 29548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:10.050756 29551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:10.051182 29450 server_base.cc:1034] running on GCE node
I20240501 02:08:10.051365 29450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:10.051600 29450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:10.052745 29450 hybrid_clock.cc:648] HybridClock initialized: now 1714529290052731 us; error 30 us; skew 500 ppm
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:10.055565 29450 init.cc:377] Logged in from keytab as oryx/127.22.12.65@KRBTEST.COM (short username oryx)
I20240501 02:08:10.056037 29450 webserver.cc:460] Webserver started at http://127.22.12.65:36641/ using document root <none> and password file <none>
I20240501 02:08:10.056233 29450 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:10.056291 29450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:10.057801 29450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:08:10.058408 29558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.058622 29450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:10.058696 29450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "1c23eec833ec4996a6d8b61316549054"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "68d9f43886092d87e6817dd19e507bb3"
server_key_iv: "77a3a566be84268838ea718bd05cc194"
server_key_version: "encryptionkey@0"
I20240501 02:08:10.059011 29450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:10.076982 29450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:10.078847 29450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:10.078966 29450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:10.079183 29450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:10.079481 29450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:10.079514 29450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.079542 29450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:10.079563 29450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.085006 29450 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:45011
I20240501 02:08:10.085063 29671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:45011 every 8 connection(s)
I20240501 02:08:10.085366 29450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.124@KRBTEST.COM
I20240501 02:08:10.092687 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 29450
I20240501 02:08:10.092840 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28897
I20240501 02:08:10.094925 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.086316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:55551 (local address 127.22.12.126:35589)
0501 02:08:10.086372 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.086374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.086463 (+ 89us) server_negotiation.cc:407] Connection header received
0501 02:08:10.086963 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.086964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.086976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.087023 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.087900 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.088646 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.089312 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.089477 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.091960 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.091965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.091965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.091972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.093830 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.094143 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.094145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.094145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.094167 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.094301 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.094302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.094302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.094402 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.094436 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.094711 (+ 275us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.094824 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:08:10.094925 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.086085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:39283 (local address 127.22.12.125:39095)
0501 02:08:10.086134 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.086136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.086477 (+ 341us) server_negotiation.cc:407] Connection header received
0501 02:08:10.086981 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.086982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.086994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.087021 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.088014 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.088472 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.089132 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.089316 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.091521 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.091525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.091526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.091534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.092982 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.093386 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.093387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.093388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.093413 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.093869 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.093871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.093872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.094004 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.094053 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.094730 (+ 677us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.094826 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:08:10.095391 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.086040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.22.12.65:39283)
0501 02:08:10.086416 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.086424 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.086879 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.087101 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.087102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.087662 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.087935 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.087936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.088554 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.088555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.089052 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.089053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.089192 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.089814 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.089831 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.091406 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.093069 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.093074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.093077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.093306 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.093479 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.093480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.093481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.093562 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.094104 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.094110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.094560 (+ 450us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.095022 (+ 462us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.095184 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":231,"thread_start_us":80,"threads_started":1}
I20240501 02:08:10.095870 29679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.086039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:55551)
0501 02:08:10.086395 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.086404 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.086864 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.087097 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.087100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.087470 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.087805 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.087809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.088724 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.088725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.089220 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.089222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.089527 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.090040 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.090045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.091844 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.093915 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.093916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.093916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.094070 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.094224 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.094225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.094225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.094249 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.094473 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.094474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.094558 (+ 84us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.095369 (+ 811us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.095704 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20240501 02:08:10.095988 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.086085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:35435 (local address 127.22.12.124:38171)
0501 02:08:10.086577 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.086580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.086592 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:08:10.087017 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.087018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.087031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.087083 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.087953 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.088910 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.089537 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.089754 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.093398 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.093404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.093404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.093411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.095016 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.095344 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.095345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.095346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.095373 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.095513 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.095514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.095514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.095619 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.095675 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.095799 (+ 124us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.095890 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":62,"threads_started":1}
I20240501 02:08:10.095994 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.086039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.22.12.65:35435)
0501 02:08:10.086266 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.086298 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.086865 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.087130 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.087131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.087473 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.087865 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.087867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.088979 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.088981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.089449 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.089451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.089524 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.090144 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.090147 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.092402 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.095096 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.095097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.095097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.095273 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.095434 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.095434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.095435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.095460 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.095668 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.095669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.095705 (+ 36us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.095770 (+ 65us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.095869 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20240501 02:08:10.096701 29672 heartbeater.cc:347] Connected to a master server at 127.22.12.124:38171
I20240501 02:08:10.096804 29672 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.097023 29672 heartbeater.cc:511] Master 127.22.12.124:38171 requested a full tablet report, sending...
I20240501 02:08:10.097537 29493 ts_manager.cc:194] Registered new tserver with Master: 1c23eec833ec4996a6d8b61316549054 (127.22.12.65:45011)
I20240501 02:08:10.099074 29674 heartbeater.cc:347] Connected to a master server at 127.22.12.125:39095
I20240501 02:08:10.099092 29673 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35589
I20240501 02:08:10.099141 29674 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.099144 29673 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.099258 29674 heartbeater.cc:511] Master 127.22.12.125:39095 requested a full tablet report, sending...
I20240501 02:08:10.099309 29673 heartbeater.cc:511] Master 127.22.12.126:35589 requested a full tablet report, sending...
I20240501 02:08:10.099560 28701 ts_manager.cc:194] Re-registered known tserver with Master: 1c23eec833ec4996a6d8b61316549054 (127.22.12.65:45011)
I20240501 02:08:10.099622 29302 ts_manager.cc:194] Registered new tserver with Master: 1c23eec833ec4996a6d8b61316549054 (127.22.12.65:45011)
I20240501 02:08:10.100031 28701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:55551
I20240501 02:08:10.100111 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:46033
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:35589
--webserver_port=41383
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:10.178730 29680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:10.178926 29680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:10.178949 29680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:10.180320 29680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:10.180378 29680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:10.180403 29680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:08:10.182025 29680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:46033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=41383
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:10.182253 29680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:10.182446 29680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:10.186995 29688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:10.187222 29686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:10.187681 29685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:10.188223 29680 server_base.cc:1034] running on GCE node
I20240501 02:08:10.188401 29680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:10.188622 29680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:10.189765 29680 hybrid_clock.cc:648] HybridClock initialized: now 1714529290189753 us; error 31 us; skew 500 ppm
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:10.192106 29680 init.cc:377] Logged in from keytab as oryx/127.22.12.66@KRBTEST.COM (short username oryx)
I20240501 02:08:10.192539 29680 webserver.cc:460] Webserver started at http://127.22.12.66:41383/ using document root <none> and password file <none>
I20240501 02:08:10.192710 29680 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:10.192765 29680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:10.193959 29680 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:10.194463 29695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.194639 29680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.194720 29680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "98108dcd65a24078925d9616fccce6ed"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "f7ab38a5bc13431e08589bac5936ae91"
server_key_iv: "fb2e77f49c3b2dd2bc8877a08f070898"
server_key_version: "encryptionkey@0"
I20240501 02:08:10.195013 29680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:10.206190 29680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:10.208097 29680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:10.208207 29680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:10.208400 29680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:10.208642 29680 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:10.208668 29680 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.208688 29680 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:10.208705 29680 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.213769 29680 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:46033
I20240501 02:08:10.213827 29808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:46033 every 8 connection(s)
I20240501 02:08:10.214128 29680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:08:10.214334 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 29680
I20240501 02:08:10.214496 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 29031
I20240501 02:08:10.220604 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:43727
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:35589
--webserver_port=46349
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.124@KRBTEST.COM
I20240501 02:08:10.225584 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.215204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:49423 (local address 127.22.12.124:38171)
0501 02:08:10.215242 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.215244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.215253 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:10.215767 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.215769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.215784 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.215841 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.217146 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.217923 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.218716 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218883 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.222816 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.222820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.222820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.222826 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.224170 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224668 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.224669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.224669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.224696 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224878 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.224879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.224880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.224994 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.225033 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.225396 (+ 363us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.225486 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:08:10.225584 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.215185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:39917 (local address 127.22.12.126:35589)
0501 02:08:10.215232 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.215234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.216631 (+ 1397us) server_negotiation.cc:407] Connection header received
0501 02:08:10.216664 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.216665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.216676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.216732 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.217323 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218056 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.218834 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.219007 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.221718 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.221725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.221725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.221733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.223423 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224669 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.224670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.224670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.224697 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224891 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.224892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.224893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.224993 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.225033 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.225391 (+ 358us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.225487 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:08:10.225584 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.215928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:42019 (local address 127.22.12.125:39095)
0501 02:08:10.215977 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.215978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.216873 (+ 895us) server_negotiation.cc:407] Connection header received
0501 02:08:10.216958 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.216959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.216968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.216993 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.217512 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218211 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.218919 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.219064 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.222512 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.222517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.222517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.222525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.224042 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224635 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.224637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.224637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.224665 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224872 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.224872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.224873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.224997 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.225039 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.225395 (+ 356us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.225486 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 02:08:10.226051 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.214896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:39917)
0501 02:08:10.216532 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.216543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.216615 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.216788 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.216789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.216985 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.217227 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.217229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218150 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.218151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.218743 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.218745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.219054 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.219957 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.219961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.221601 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.224148 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.224150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.224583 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.224763 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.224764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.224815 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.225076 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.225081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.225229 (+ 148us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.225734 (+ 505us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.225916 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1614,"mutex_wait_us":103,"thread_start_us":81,"threads_started":1}
I20240501 02:08:10.226051 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.214892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.22.12.66:49423)
0501 02:08:10.215044 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.215072 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.215644 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.215904 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.215909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.216319 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.217046 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.217050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218028 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.218028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.218626 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.218628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218987 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.219761 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.219785 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.222735 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.224223 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224223 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.224224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.224583 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.224763 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.224764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.224815 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.225076 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.225081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.225229 (+ 148us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.225681 (+ 452us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.225850 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20240501 02:08:10.226506 29810 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35589
I20240501 02:08:10.226586 29809 heartbeater.cc:347] Connected to a master server at 127.22.12.124:38171
I20240501 02:08:10.226665 29810 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.226711 29809 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.226887 29810 heartbeater.cc:511] Master 127.22.12.126:35589 requested a full tablet report, sending...
I20240501 02:08:10.226898 29809 heartbeater.cc:511] Master 127.22.12.124:38171 requested a full tablet report, sending...
I20240501 02:08:10.227155 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.215451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.22.12.66:42019)
0501 02:08:10.216817 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.216826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.216896 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.217050 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.217051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.217211 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.217428 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.217430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.218289 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.218289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.218839 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.218841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.219107 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.219811 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.219814 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.222408 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.224135 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.224145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.224541 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.224729 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.224729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.224730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.224809 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.225082 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.225083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.225234 (+ 151us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.226397 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.226504 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"thread_start_us":55,"threads_started":1}
I20240501 02:08:10.227295 28701 ts_manager.cc:194] Re-registered known tserver with Master: 98108dcd65a24078925d9616fccce6ed (127.22.12.66:46033)
I20240501 02:08:10.227325 29493 ts_manager.cc:194] Registered new tserver with Master: 98108dcd65a24078925d9616fccce6ed (127.22.12.66:46033)
I20240501 02:08:10.227684 28701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:39917
I20240501 02:08:10.227780 29811 heartbeater.cc:347] Connected to a master server at 127.22.12.125:39095
I20240501 02:08:10.227821 29811 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.227900 29811 heartbeater.cc:511] Master 127.22.12.125:39095 requested a full tablet report, sending...
I20240501 02:08:10.228204 29302 ts_manager.cc:194] Registered new tserver with Master: 98108dcd65a24078925d9616fccce6ed (127.22.12.66:46033)
W20240501 02:08:10.301213 29816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:10.301414 29816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:10.301436 29816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:10.302811 29816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:10.302857 29816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:10.302882 29816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:08:10.304524 29816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:43727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=46349
--tserver_master_addrs=127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:10.304734 29816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:10.304925 29816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:10.309487 29824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:10.309875 29822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:10.310340 29821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:10.310734 29816 server_base.cc:1034] running on GCE node
I20240501 02:08:10.310911 29816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:10.311117 29816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:10.312275 29816 hybrid_clock.cc:648] HybridClock initialized: now 1714529290312232 us; error 54 us; skew 500 ppm
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:10.314759 29816 init.cc:377] Logged in from keytab as oryx/127.22.12.67@KRBTEST.COM (short username oryx)
I20240501 02:08:10.315155 29816 webserver.cc:460] Webserver started at http://127.22.12.67:46349/ using document root <none> and password file <none>
I20240501 02:08:10.315318 29816 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:10.315369 29816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:10.316612 29816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:10.317106 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.317265 29816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:10.317337 29816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "878c3f3a84b7499389ce87ce2e75df74"
format_stamp: "Formatted at 2024-05-01 02:08:09 on dist-test-slave-9tkh"
server_key: "1cccbee1655855b532c1697ba236ea91"
server_key_iv: "5740863fa93b4e92bc6b25cbf786d1bd"
server_key_version: "encryptionkey@0"
I20240501 02:08:10.317625 29816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:10.341311 29816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:10.343400 29816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:10.343499 29816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:10.343681 29816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:10.343923 29816 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:10.343950 29816 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.343971 29816 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:10.343984 29816 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:10.349246 29816 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:43727
I20240501 02:08:10.349304 29944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:43727 every 8 connection(s)
I20240501 02:08:10.349586 29816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.124@KRBTEST.COM
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
May 01 02:08:10 dist-test-slave-9tkh krb5kdc[28656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529290, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.125@KRBTEST.COM
I20240501 02:08:10.356295 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 29816
I20240501 02:08:10.358785 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.350468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:34739 (local address 127.22.12.124:38171)
0501 02:08:10.350598 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.350600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.350670 (+ 70us) server_negotiation.cc:407] Connection header received
0501 02:08:10.351032 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.351033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.351044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.351099 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.352194 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.352671 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.353257 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353432 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.355473 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.355478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.355479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.355486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.357203 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.357821 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.357823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.357823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.357848 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358055 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.358055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.358056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.358162 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.358202 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.358607 (+ 405us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.358695 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20240501 02:08:10.359067 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.350230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:50185 (local address 127.22.12.125:39095)
0501 02:08:10.350282 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.350284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.351152 (+ 868us) server_negotiation.cc:407] Connection header received
0501 02:08:10.351236 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.351237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.351247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.351288 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.352173 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.352930 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.353599 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353764 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.356534 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.356539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.356540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.356560 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.358052 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358392 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.358393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.358394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.358417 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358583 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.358583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.358584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.358679 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.358717 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.358909 (+ 192us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.358987 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 02:08:10.359367 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.350210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:39095 (local address 127.22.12.67:50185)
0501 02:08:10.351095 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.351103 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.351173 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.351301 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.351302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.351814 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.352086 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.352090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353013 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.353014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.353521 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.353523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353587 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.354274 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.354277 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.356440 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.358167 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.358168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.358322 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.358474 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.358475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.358499 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.358758 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.358759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.358793 (+ 34us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.359140 (+ 347us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.359251 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":86,"thread_start_us":83,"threads_started":1}
I20240501 02:08:10.359376 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.350199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:38171 (local address 127.22.12.67:34739)
0501 02:08:10.350602 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.350611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.350963 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.351168 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.351172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.351684 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.352086 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.352092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.352746 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.352747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.353181 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.353182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353308 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.353902 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.353917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.355389 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.357303 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.357308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.357311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.357731 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.357914 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.357914 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.357915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.357992 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.358241 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.358246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.358403 (+ 157us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.359022 (+ 619us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.359161 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20240501 02:08:10.359439 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.350468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:45321 (local address 127.22.12.126:35589)
0501 02:08:10.350547 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.350549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.350559 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:08:10.351052 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.351054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.351064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.351114 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.352201 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.352941 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.353708 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353886 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.356246 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.356250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.356251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.356256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.358416 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358777 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.358778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.358778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.358801 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358951 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.358952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.358952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.359049 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.359092 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.359275 (+ 183us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.359349 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:08:10.359809 29945 heartbeater.cc:347] Connected to a master server at 127.22.12.124:38171
I20240501 02:08:10.359905 29945 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.360146 29945 heartbeater.cc:511] Master 127.22.12.124:38171 requested a full tablet report, sending...
I20240501 02:08:10.359902 29947 heartbeater.cc:347] Connected to a master server at 127.22.12.125:39095
I20240501 02:08:10.360342 29947 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.360410 29947 heartbeater.cc:511] Master 127.22.12.125:39095 requested a full tablet report, sending...
I20240501 02:08:10.360667 29302 ts_manager.cc:194] Registered new tserver with Master: 878c3f3a84b7499389ce87ce2e75df74 (127.22.12.67:43727)
I20240501 02:08:10.360713 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.350199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:45321)
0501 02:08:10.350413 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.350442 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.350949 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.351169 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.351173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.351681 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:10.352086 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.352092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353039 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.353040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.353473 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.353475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.353535 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.354090 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:10.354094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:10.356143 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:10.358511 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.358512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.358697 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.358864 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:10.358865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:10.358865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:10.358895 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:10.359119 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:10.359119 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:10.359158 (+ 39us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.360467 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.360627 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20240501 02:08:10.361068 29946 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35589
I20240501 02:08:10.361124 29946 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:10.361160 29493 ts_manager.cc:194] Registered new tserver with Master: 878c3f3a84b7499389ce87ce2e75df74 (127.22.12.67:43727)
I20240501 02:08:10.361220 29946 heartbeater.cc:511] Master 127.22.12.126:35589 requested a full tablet report, sending...
I20240501 02:08:10.361450 28701 ts_manager.cc:194] Re-registered known tserver with Master: 878c3f3a84b7499389ce87ce2e75df74 (127.22.12.67:43727)
I20240501 02:08:10.361948 28701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:45321
I20240501 02:08:10.366575 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.357406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.22.12.126:35589)
0501 02:08:10.359518 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.359520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.359528 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:10.359546 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.359546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.359556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.359615 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.360245 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.360908 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.362011 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.362186 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.364010 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.364013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.364013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.364020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.365527 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.365822 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.365823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.365824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.365847 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.365998 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.365999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.365999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.366097 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.366174 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.366380 (+ 206us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.366456 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2097,"thread_start_us":84,"threads_started":1}
I20240501 02:08:10.367162 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.357401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.22.12.124:38171)
0501 02:08:10.358277 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.358279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.358416 (+ 137us) server_negotiation.cc:407] Connection header received
0501 02:08:10.358436 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.358437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.358449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.358496 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.361756 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.362511 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.363135 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.363288 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.364660 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.364664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.364664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.364670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.366064 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.366335 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.366336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.366337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.366360 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.366511 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.366512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.366513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.366634 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.366721 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.367008 (+ 287us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.367092 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":56,"threads_started":1}
I20240501 02:08:10.368258 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.357883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.22.12.125:39095)
0501 02:08:10.358230 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.358232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.361311 (+ 3079us) server_negotiation.cc:407] Connection header received
0501 02:08:10.361425 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.361426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.361437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.361548 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.362906 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.363655 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.364192 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.364357 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.365514 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.365519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.365519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.365525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.367137 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.367661 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.367661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.367662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.367685 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.367829 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.367830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.367830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.367919 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.368020 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.368085 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.368169 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":51,"threads_started":1}
I20240501 02:08:10.368377 28701 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:08:10.369756 28701 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:08:10.376258 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.373119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45011 (local address 127.0.0.1:52184)
0501 02:08:10.373241 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.373251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.373290 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.374304 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.374305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.374307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:10.374453 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.374454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.375307 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.375308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.376004 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.376006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.376071 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.376075 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.376102 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.376136 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20240501 02:08:10.376595 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.373120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:46033 (local address 127.0.0.1:55696)
0501 02:08:10.373360 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.373369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.373493 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.373768 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.373769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.373771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:10.373965 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.373967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.375141 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.375142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.376027 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.376029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.376090 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.376093 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.376119 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.376159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20240501 02:08:10.376681 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.373179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:43727 (local address 127.0.0.1:58038)
0501 02:08:10.374048 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.374055 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.374092 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.374282 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.374283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.374285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:10.374474 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.374476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.375704 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.375705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.376508 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.376510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.376567 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.376570 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.376594 (+ 24us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.376631 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":91,"threads_started":1}
I20240501 02:08:10.376905 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.373388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.22.12.65:45011)
0501 02:08:10.374166 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.374172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.374178 (+ 6us) server_negotiation.cc:407] Connection header received
0501 02:08:10.374212 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.374214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.374227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.374291 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:10.374516 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.375231 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.376203 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.376701 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.376759 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.376794 (+ 35us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.376837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":76,"threads_started":1}
I20240501 02:08:10.377413 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.373379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.22.12.67:43727)
0501 02:08:10.373531 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.373537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.374129 (+ 592us) server_negotiation.cc:407] Connection header received
0501 02:08:10.374170 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.374172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.374188 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.374576 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:10.374642 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.375613 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.376850 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.377266 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.377298 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.377323 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.377351 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20240501 02:08:10.377519 29606 tablet_service.cc:1467] Processing CreateTablet for tablet 828515fcb5e14d329b10bcd710749826 (DEFAULT_TABLE table=test-table [id=335b2ac601934e85a611d16094d55882]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:10.377687 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.373203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.22.12.66:46033)
0501 02:08:10.373373 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.373384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.373546 (+ 162us) server_negotiation.cc:407] Connection header received
0501 02:08:10.373590 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.373592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.373610 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.373682 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:10.374027 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.375065 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.377023 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.377508 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.377551 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.377583 (+ 32us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.377617 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20240501 02:08:10.377826 29606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828515fcb5e14d329b10bcd710749826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:10.377902 29879 tablet_service.cc:1467] Processing CreateTablet for tablet 828515fcb5e14d329b10bcd710749826 (DEFAULT_TABLE table=test-table [id=335b2ac601934e85a611d16094d55882]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:10.378121 29879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828515fcb5e14d329b10bcd710749826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:10.378197 29743 tablet_service.cc:1467] Processing CreateTablet for tablet 828515fcb5e14d329b10bcd710749826 (DEFAULT_TABLE table=test-table [id=335b2ac601934e85a611d16094d55882]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:10.378446 29743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828515fcb5e14d329b10bcd710749826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:10.380868 29971 tablet_bootstrap.cc:492] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Bootstrap starting.
I20240501 02:08:10.381310 29972 tablet_bootstrap.cc:492] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Bootstrap starting.
I20240501 02:08:10.381611 29973 tablet_bootstrap.cc:492] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Bootstrap starting.
I20240501 02:08:10.381870 29972 tablet_bootstrap.cc:654] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:10.381955 29971 tablet_bootstrap.cc:654] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:10.382169 29973 tablet_bootstrap.cc:654] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:10.382159 29972 log.cc:824] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:10.382297 29971 log.cc:824] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:10.382454 29973 log.cc:824] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:10.382917 29972 tablet_bootstrap.cc:492] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: No bootstrap required, opened a new log
I20240501 02:08:10.382984 29972 ts_tablet_manager.cc:1367] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:08:10.383050 29971 tablet_bootstrap.cc:492] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: No bootstrap required, opened a new log
I20240501 02:08:10.383119 29973 tablet_bootstrap.cc:492] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: No bootstrap required, opened a new log
I20240501 02:08:10.383121 29971 ts_tablet_manager.cc:1367] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:08:10.383190 29973 ts_tablet_manager.cc:1367] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:08:10.384389 29972 raft_consensus.cc:348] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.384505 29972 raft_consensus.cc:374] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:10.384526 29972 raft_consensus.cc:729] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c23eec833ec4996a6d8b61316549054, State: Initialized, Role: FOLLOWER
I20240501 02:08:10.384613 29972 consensus_queue.cc:260] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.384825 29972 ts_tablet_manager.cc:1398] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:08:10.384873 29973 raft_consensus.cc:348] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.384845 29971 raft_consensus.cc:348] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.384989 29971 raft_consensus.cc:374] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:10.384989 29973 raft_consensus.cc:374] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:10.385011 29973 raft_consensus.cc:729] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98108dcd65a24078925d9616fccce6ed, State: Initialized, Role: FOLLOWER
I20240501 02:08:10.385011 29971 raft_consensus.cc:729] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878c3f3a84b7499389ce87ce2e75df74, State: Initialized, Role: FOLLOWER
I20240501 02:08:10.385099 29673 heartbeater.cc:503] Master 127.22.12.126:35589 was elected leader, sending a full tablet report...
I20240501 02:08:10.385094 29971 consensus_queue.cc:260] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.385313 29971 ts_tablet_manager.cc:1398] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:08:10.385576 29946 heartbeater.cc:503] Master 127.22.12.126:35589 was elected leader, sending a full tablet report...
I20240501 02:08:10.385509 29973 consensus_queue.cc:260] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.385859 29973 ts_tablet_manager.cc:1398] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 02:08:10.385946 29810 heartbeater.cc:503] Master 127.22.12.126:35589 was elected leader, sending a full tablet report...
I20240501 02:08:10.425578 29978 raft_consensus.cc:482] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:10.425714 29978 raft_consensus.cc:504] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.426061 29978 leader_election.cc:288] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c23eec833ec4996a6d8b61316549054 (127.22.12.65:45011), 98108dcd65a24078925d9616fccce6ed (127.22.12.66:46033)
I20240501 02:08:10.428144 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.426227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:46033 (local address 127.22.12.67:57763)
0501 02:08:10.426308 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.426319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.426358 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.426501 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.426501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.426503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:10.426687 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.426689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.427299 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.427299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.427870 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.427872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.427932 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.427937 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.427963 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.428000 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":46}
I20240501 02:08:10.428144 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.426218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45011 (local address 127.22.12.67:50279)
0501 02:08:10.426268 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:10.426280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:10.426323 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:10.426452 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:10.426453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:10.426455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:10.426658 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.426660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.427280 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.427281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:10.427873 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:10.427874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.427939 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.427943 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:08:10.427969 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:08:10.428001 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20240501 02:08:10.428452 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.426281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:57763 (local address 127.22.12.66:46033)
0501 02:08:10.426333 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.426335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.426347 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:08:10.426428 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.426429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.426438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.426489 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:10.426762 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.427225 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.427954 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.428360 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.428368 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.428386 (+ 18us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.428408 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:08:10.428474 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.426288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:50279 (local address 127.22.12.65:45011)
0501 02:08:10.426329 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.426331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.426342 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:08:10.426381 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.426382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.426391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.426444 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:10.426723 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.427207 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.427953 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.428383 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.428390 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.428407 (+ 17us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.428430 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:08:10.428721 29763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828515fcb5e14d329b10bcd710749826" candidate_uuid: "878c3f3a84b7499389ce87ce2e75df74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98108dcd65a24078925d9616fccce6ed" is_pre_election: true
I20240501 02:08:10.428730 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828515fcb5e14d329b10bcd710749826" candidate_uuid: "878c3f3a84b7499389ce87ce2e75df74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c23eec833ec4996a6d8b61316549054" is_pre_election: true
I20240501 02:08:10.428869 29626 raft_consensus.cc:2454] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878c3f3a84b7499389ce87ce2e75df74 in term 0.
I20240501 02:08:10.428867 29763 raft_consensus.cc:2454] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878c3f3a84b7499389ce87ce2e75df74 in term 0.
I20240501 02:08:10.429042 29833 leader_election.cc:302] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878c3f3a84b7499389ce87ce2e75df74, 98108dcd65a24078925d9616fccce6ed; no voters:
I20240501 02:08:10.429198 29978 raft_consensus.cc:2781] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:08:10.429280 29978 raft_consensus.cc:482] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:08:10.429311 29978 raft_consensus.cc:3037] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:10.430123 29978 raft_consensus.cc:504] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.430253 29978 leader_election.cc:288] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [CANDIDATE]: Term 1 election: Requested vote from peers 1c23eec833ec4996a6d8b61316549054 (127.22.12.65:45011), 98108dcd65a24078925d9616fccce6ed (127.22.12.66:46033)
I20240501 02:08:10.430423 29763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828515fcb5e14d329b10bcd710749826" candidate_uuid: "878c3f3a84b7499389ce87ce2e75df74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98108dcd65a24078925d9616fccce6ed"
I20240501 02:08:10.430429 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828515fcb5e14d329b10bcd710749826" candidate_uuid: "878c3f3a84b7499389ce87ce2e75df74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c23eec833ec4996a6d8b61316549054"
I20240501 02:08:10.430500 29626 raft_consensus.cc:3037] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:10.430501 29763 raft_consensus.cc:3037] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:10.431401 29763 raft_consensus.cc:2454] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878c3f3a84b7499389ce87ce2e75df74 in term 1.
I20240501 02:08:10.431555 29833 leader_election.cc:302] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878c3f3a84b7499389ce87ce2e75df74, 98108dcd65a24078925d9616fccce6ed; no voters:
I20240501 02:08:10.431649 29626 raft_consensus.cc:2454] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878c3f3a84b7499389ce87ce2e75df74 in term 1.
I20240501 02:08:10.431668 29978 raft_consensus.cc:2781] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:10.431876 29978 raft_consensus.cc:686] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 1 LEADER]: Becoming Leader. State: Replica: 878c3f3a84b7499389ce87ce2e75df74, State: Running, Role: LEADER
I20240501 02:08:10.431985 29978 consensus_queue.cc:237] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } }
I20240501 02:08:10.432711 28701 catalog_manager.cc:5484] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 878c3f3a84b7499389ce87ce2e75df74 (127.22.12.67). New cstate: current_term: 1 leader_uuid: "878c3f3a84b7499389ce87ce2e75df74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878c3f3a84b7499389ce87ce2e75df74" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43727 } health_report { overall_health: HEALTHY } } }
I20240501 02:08:10.449769 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.447968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.22.12.67:43727)
0501 02:08:10.448024 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.448026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.448034 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:10.448103 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.448104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.448120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.448173 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:10.448383 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.448835 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.449344 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.449466 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.449503 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:10.449587 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:10.449646 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.449677 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.449695 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:08:10.451416 29763 raft_consensus.cc:1261] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 1 FOLLOWER]: Refusing update from remote peer 878c3f3a84b7499389ce87ce2e75df74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:10.451421 29626 raft_consensus.cc:1261] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 1 FOLLOWER]: Refusing update from remote peer 878c3f3a84b7499389ce87ce2e75df74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:10.451709 29978 consensus_queue.cc:1035] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c23eec833ec4996a6d8b61316549054" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:10.451840 29980 consensus_queue.cc:1035] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98108dcd65a24078925d9616fccce6ed" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:10.454080 29983 mvcc.cc:205] Tried to move back new op lower bound from 7022711973686501376 to 7022711973609963520. Current Snapshot: MvccSnapshot[applied={T|T < 7022711973686501376}]
I20240501 02:08:10.454259 29985 mvcc.cc:205] Tried to move back new op lower bound from 7022711973686501376 to 7022711973609963520. Current Snapshot: MvccSnapshot[applied={T|T < 7022711973686501376}]
I20240501 02:08:10.456853 29984 mvcc.cc:205] Tried to move back new op lower bound from 7022711973686501376 to 7022711973609963520. Current Snapshot: MvccSnapshot[applied={T|T < 7022711973686501376}]
I20240501 02:08:10.456897 28701 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36870:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:10.456983 28701 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36870:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:10.458195 28701 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 34b24cf1f77b46c6a5ff8821786121c7: Sending DeleteTablet for 3 replicas of tablet 828515fcb5e14d329b10bcd710749826
I20240501 02:08:10.458570 29743 tablet_service.cc:1514] Processing DeleteTablet for tablet 828515fcb5e14d329b10bcd710749826 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:10 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:55696
I20240501 02:08:10.458603 29879 tablet_service.cc:1514] Processing DeleteTablet for tablet 828515fcb5e14d329b10bcd710749826 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:10 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:58038
I20240501 02:08:10.458774 29993 tablet_replica.cc:330] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: stopping tablet replica
I20240501 02:08:10.458829 29994 tablet_replica.cc:330] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: stopping tablet replica
I20240501 02:08:10.458861 29993 raft_consensus.cc:2229] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:10.458899 29994 raft_consensus.cc:2229] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:08:10.459077 29606 tablet_service.cc:1514] Processing DeleteTablet for tablet 828515fcb5e14d329b10bcd710749826 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:10 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:52184
I20240501 02:08:10.458959 29993 raft_consensus.cc:2258] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:10.459326 29995 tablet_replica.cc:330] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: stopping tablet replica
I20240501 02:08:10.459458 29995 raft_consensus.cc:2229] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:10.459595 29995 raft_consensus.cc:2258] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:10.458982 29994 raft_consensus.cc:2258] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:10.459892 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 28669
I20240501 02:08:10.460144 29995 ts_tablet_manager.cc:1875] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:10.460776 29994 ts_tablet_manager.cc:1875] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:10.462025 29995 ts_tablet_manager.cc:1888] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:10.462152 29995 log.cc:1196] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/828515fcb5e14d329b10bcd710749826
I20240501 02:08:10.462245 29994 ts_tablet_manager.cc:1888] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:10.462311 29994 log.cc:1196] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/828515fcb5e14d329b10bcd710749826
I20240501 02:08:10.462568 29994 ts_tablet_manager.cc:1909] T 828515fcb5e14d329b10bcd710749826 P 878c3f3a84b7499389ce87ce2e75df74: Deleting consensus metadata
I20240501 02:08:10.462807 29995 ts_tablet_manager.cc:1909] T 828515fcb5e14d329b10bcd710749826 P 1c23eec833ec4996a6d8b61316549054: Deleting consensus metadata
W20240501 02:08:10.466516 29835 connection.cc:535] server connection from 127.0.0.1:58038 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 02:08:10.468281 29993 ts_tablet_manager.cc:1875] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:10.470131 29993 ts_tablet_manager.cc:1888] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:10.470211 29993 log.cc:1196] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/828515fcb5e14d329b10bcd710749826
I20240501 02:08:10.470506 29993 ts_tablet_manager.cc:1909] T 828515fcb5e14d329b10bcd710749826 P 98108dcd65a24078925d9616fccce6ed: Deleting consensus metadata
W20240501 02:08:10.470850 29697 connection.cc:441] server connection from 127.0.0.1:55696 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55696 (request call id 1) could send its response
I20240501 02:08:10.471794 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.459420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.22.12.124:38171)
0501 02:08:10.459461 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.459464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.463022 (+ 3558us) server_negotiation.cc:407] Connection header received
0501 02:08:10.463117 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.463118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.463132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.463158 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.464076 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.464841 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.465616 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.465803 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.467881 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.467895 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.467895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.467903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.470775 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.471108 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.471110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.471111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.471143 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.471278 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.471278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.471279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.471380 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.471463 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.471622 (+ 159us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.471708 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:08:10.477023 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.468281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.22.12.125:39095)
0501 02:08:10.468325 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.468327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.469257 (+ 930us) server_negotiation.cc:407] Connection header received
0501 02:08:10.469812 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.469813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.469828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.469896 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.472574 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.473094 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.473702 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.474136 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.474752 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.474755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.474756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.474761 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.476145 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.476417 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.476417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.476418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.476439 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.476551 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.476552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.476553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.476642 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.476718 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.476835 (+ 117us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.476927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:08:10.477023 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:10.468659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.22.12.124:38171)
0501 02:08:10.468710 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:10.468712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:10.470056 (+ 1344us) server_negotiation.cc:407] Connection header received
0501 02:08:10.470137 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:10.470138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:10.470151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:10.470211 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:10.471423 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.472225 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:10.473005 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:10.473170 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:10.474391 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:10.474396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:10.474396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:10.474402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:10.475936 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.476279 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.476280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.476280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.476300 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:10.476465 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:10.476467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:10.476467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:10.476577 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:10.476649 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:10.476836 (+ 187us) server_negotiation.cc:299] Negotiation successful
0501 02:08:10.476925 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:08:11.455595 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:11.455345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:37271)
0501 02:08:11.455479 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:11.455540 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
W20240501 02:08:11.455814 29946 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:35589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:11.455994 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:11.455794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:48925)
0501 02:08:11.455897 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:11.455951 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
W20240501 02:08:11.456115 29810 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:35589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:11.457180 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:11.456979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:52433)
0501 02:08:11.457076 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:11.457132 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
W20240501 02:08:11.457305 29673 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:35589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:11.958794 30007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34b24cf1f77b46c6a5ff8821786121c7)
I20240501 02:08:11.958935 30007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:11.959156 30007 leader_election.cc:288] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589)
I20240501 02:08:11.959468 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:11.959288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:37362)
0501 02:08:11.959377 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:11.959429 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":39,"threads_started":1}
W20240501 02:08:11.959570 29287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:11.959951 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:11.959851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:37372)
0501 02:08:11.959894 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:11.959931 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240501 02:08:11.960027 29287 leader_election.cc:334] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589): Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:11.960074 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2cab249f87cf4277a357ab25503066b9; no voters: 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:11.960197 30007 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:08:12.456422 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:12.456156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:51575)
0501 02:08:12.456312 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:12.456360 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20240501 02:08:12.456626 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:12.456416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:60783)
0501 02:08:12.456553 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:12.456587 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":45,"threads_started":1}
I20240501 02:08:12.457772 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:12.457565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:37695)
0501 02:08:12.457686 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:12.457720 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":54,"threads_started":1}
I20240501 02:08:13.457126 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:13.456882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:59125)
0501 02:08:13.457026 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:13.457066 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20240501 02:08:13.457448 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:13.457124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:46505)
0501 02:08:13.457367 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:13.457405 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":49,"threads_started":1}
I20240501 02:08:13.458343 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:13.458190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:58843)
0501 02:08:13.458276 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:13.458306 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":44,"threads_started":1}
I20240501 02:08:13.814901 30016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34b24cf1f77b46c6a5ff8821786121c7)
I20240501 02:08:13.815012 30016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:13.815158 30016 leader_election.cc:288] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589)
I20240501 02:08:13.815491 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:13.815295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52086)
0501 02:08:13.815413 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:13.815449 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20240501 02:08:13.815840 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:13.815752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52088)
0501 02:08:13.815794 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:13.815817 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240501 02:08:13.815917 29287 leader_election.cc:334] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589): Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:13.815953 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2cab249f87cf4277a357ab25503066b9; no voters: 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:13.816012 30016 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:08:14.457747 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:14.457483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:48319)
0501 02:08:14.457632 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:14.457689 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20240501 02:08:14.458071 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:14.457793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:54571)
0501 02:08:14.457992 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:14.458025 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":44,"threads_started":1}
I20240501 02:08:14.458812 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:14.458660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:60705)
0501 02:08:14.458747 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:14.458777 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":42,"threads_started":1}
I20240501 02:08:15.458685 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:15.458439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:36803)
0501 02:08:15.458579 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:15.458625 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20240501 02:08:15.458918 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:15.458686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:50151)
0501 02:08:15.458835 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:15.458869 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":47,"threads_started":1}
I20240501 02:08:15.459345 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:15.459156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:57495)
0501 02:08:15.459281 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:15.459311 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20240501 02:08:15.839679 30025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34b24cf1f77b46c6a5ff8821786121c7)
I20240501 02:08:15.839789 30025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:15.839958 30025 leader_election.cc:288] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589)
I20240501 02:08:15.840301 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:15.840112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52304)
0501 02:08:15.840223 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:15.840259 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20240501 02:08:15.840705 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:15.840618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52318)
0501 02:08:15.840662 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:15.840686 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:08:15.840790 29287 leader_election.cc:334] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589): Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:15.840853 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2cab249f87cf4277a357ab25503066b9; no voters: 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:15.840925 30025 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:08:16.459511 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:16.459281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:52567)
0501 02:08:16.459406 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:16.459439 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20240501 02:08:16.459503 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:16.459266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:37461)
0501 02:08:16.459396 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:16.459433 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20240501 02:08:16.459827 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:16.459664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:44059)
0501 02:08:16.459757 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:16.459786 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
I20240501 02:08:17.460182 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:17.459970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:47985)
0501 02:08:17.460083 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:17.460120 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20240501 02:08:17.460230 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:17.459964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:34789)
0501 02:08:17.460133 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:17.460181 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":62,"threads_started":1}
I20240501 02:08:17.460371 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:17.460096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:36855)
0501 02:08:17.460188 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:17.460218 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":45,"threads_started":1}
I20240501 02:08:17.621372 30034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34b24cf1f77b46c6a5ff8821786121c7)
I20240501 02:08:17.621523 30034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:17.621706 30034 leader_election.cc:288] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589)
I20240501 02:08:17.622078 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:17.621851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52488)
0501 02:08:17.621988 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:17.622032 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
W20240501 02:08:17.622180 29287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240501 02:08:17.622582 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:17.622458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52504)
0501 02:08:17.622502 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:17.622558 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20240501 02:08:17.622707 29287 leader_election.cc:334] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589): Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:17.622782 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2cab249f87cf4277a357ab25503066b9; no voters: 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:17.622853 30034 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:08:18.460847 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:18.460599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:38381)
0501 02:08:18.460747 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:18.460789 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20240501 02:08:18.461004 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:18.460602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:56349)
0501 02:08:18.460860 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:18.460921 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":66,"threads_started":1}
I20240501 02:08:18.461599 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:18.460817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:43037)
0501 02:08:18.460937 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:18.461539 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":44,"threads_started":1}
I20240501 02:08:19.461624 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:19.461393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:50937)
0501 02:08:19.461532 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:19.461579 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20240501 02:08:19.461753 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:19.461569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:35439)
0501 02:08:19.461673 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:19.461704 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20240501 02:08:19.462117 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:19.461971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:34585)
0501 02:08:19.462057 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:19.462085 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":43,"threads_started":1}
I20240501 02:08:20.462297 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:20.462096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:43941)
0501 02:08:20.462215 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:20.462254 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20240501 02:08:20.462298 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:20.462086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:41833)
0501 02:08:20.462212 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:20.462251 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240501 02:08:20.462633 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:20.462466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:57371)
0501 02:08:20.462573 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:20.462602 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":49,"threads_started":1}
I20240501 02:08:21.462970 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:21.462721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:32793)
0501 02:08:21.462886 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:21.462914 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20240501 02:08:21.462970 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:21.462711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:34641)
0501 02:08:21.462875 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:21.462914 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20240501 02:08:21.463219 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:21.463035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:51339)
0501 02:08:21.463149 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:21.463181 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20240501 02:08:22.463649 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:22.463381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:45755)
0501 02:08:22.463547 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:22.463587 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":38,"threads_started":1}
I20240501 02:08:22.463747 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:22.463335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:34267)
0501 02:08:22.463679 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:22.463707 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":76,"threads_started":1}
I20240501 02:08:22.463784 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:22.463601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:48837)
0501 02:08:22.463700 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:22.463733 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
I20240501 02:08:23.030097 30052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34b24cf1f77b46c6a5ff8821786121c7)
I20240501 02:08:23.030205 30052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:23.030388 30052 leader_election.cc:288] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589)
I20240501 02:08:23.030794 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:23.030620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52910)
0501 02:08:23.030716 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.030753 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":48,"threads_started":1}
W20240501 02:08:23.030879 29287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:23.031292 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:23.031170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:52916)
0501 02:08:23.031239 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.031269 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20240501 02:08:23.031394 29287 leader_election.cc:334] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589): Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:23.031453 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2cab249f87cf4277a357ab25503066b9; no voters: 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:23.031536 30052 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:08:23.464263 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:23.464016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:46067)
0501 02:08:23.464167 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.464206 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20240501 02:08:23.464429 30057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:23.464262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:42035)
0501 02:08:23.464362 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.464384 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":37,"threads_started":1}
I20240501 02:08:23.464429 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:23.464111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:37503)
0501 02:08:23.464349 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.464382 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":55,"threads_started":1}
I20240501 02:08:24.465027 30058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:24.464807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:38969)
0501 02:08:24.464930 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.464967 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20240501 02:08:24.465091 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:24.464896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:52543)
0501 02:08:24.465010 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.465042 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20240501 02:08:24.465173 30059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:24.464811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:41113)
0501 02:08:24.465100 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.465133 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":59,"threads_started":1}
I20240501 02:08:25.465826 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:25.465613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:40689)
0501 02:08:25.465733 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:25.465769 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20240501 02:08:25.465855 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:25.465667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:50305)
0501 02:08:25.465774 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:25.465805 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20240501 02:08:25.465975 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:25.465761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:39015)
0501 02:08:25.465851 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:25.465874 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
I20240501 02:08:26.466583 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:26.466229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:59415)
0501 02:08:26.466459 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:26.466500 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20240501 02:08:26.466595 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:26.466339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:51363)
0501 02:08:26.466487 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:26.466520 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":54,"threads_started":1}
I20240501 02:08:26.466634 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:26.466269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:45563)
0501 02:08:26.466565 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:26.466594 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":53,"threads_started":1}
I20240501 02:08:27.467274 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:27.467014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:42849)
0501 02:08:27.467185 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:27.467218 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20240501 02:08:27.467422 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:27.467014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:60587)
0501 02:08:27.467180 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:27.467218 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20240501 02:08:27.467432 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:27.467158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:34925)
0501 02:08:27.467344 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:27.467380 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":55,"threads_started":1}
I20240501 02:08:28.467979 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:28.467724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:38301)
0501 02:08:28.467883 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:28.467923 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20240501 02:08:28.468006 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:28.467766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:51839)
0501 02:08:28.467915 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:28.467954 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":43,"threads_started":1}
I20240501 02:08:28.468077 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:28.467787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:48935)
0501 02:08:28.467994 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:28.468023 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":54,"threads_started":1}
I20240501 02:08:29.468658 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:29.468435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:35959)
0501 02:08:29.468560 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:29.468598 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240501 02:08:29.468744 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:29.468547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:47809)
0501 02:08:29.468670 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:29.468698 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":52,"threads_started":1}
I20240501 02:08:29.468744 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:29.468547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:59089)
0501 02:08:29.468666 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:29.468698 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20240501 02:08:30.469314 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:30.469098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:44539)
0501 02:08:30.469218 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:30.469255 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20240501 02:08:30.469419 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:30.469164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:38637)
0501 02:08:30.469321 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:30.469351 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":38,"threads_started":1}
I20240501 02:08:30.469527 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:30.469129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:57359)
0501 02:08:30.469444 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:30.469478 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":56,"threads_started":1}
I20240501 02:08:30.835697 30079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34b24cf1f77b46c6a5ff8821786121c7)
I20240501 02:08:30.835812 30079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "34b24cf1f77b46c6a5ff8821786121c7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35589 } } peers { permanent_uuid: "2cab249f87cf4277a357ab25503066b9" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 39095 } attrs { promote: false } } peers { permanent_uuid: "7a6f9aecf5a24a97b897b31ef3f0058c" member_type: NON_VOTER last_known_addr { host: "127.22.12.124" port: 38171 } attrs { promote: true } }
I20240501 02:08:30.836000 30079 leader_election.cc:288] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589)
I20240501 02:08:30.836359 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:30.836158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:38728)
0501 02:08:30.836279 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:30.836317 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20240501 02:08:30.836442 29287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:30.836802 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:30.836711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.0.0.1:38740)
0501 02:08:30.836757 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:30.836782 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240501 02:08:30.836903 29287 leader_election.cc:334] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34b24cf1f77b46c6a5ff8821786121c7 (127.22.12.126:35589): Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
I20240501 02:08:30.836966 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2cab249f87cf4277a357ab25503066b9; no voters: 34b24cf1f77b46c6a5ff8821786121c7
I20240501 02:08:30.837050 30079 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cab249f87cf4277a357ab25503066b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:08:31.470085 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:31.469833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:44027)
0501 02:08:31.470017 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:31.470043 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
I20240501 02:08:31.470085 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:31.469836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:60265)
0501 02:08:31.469988 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:31.470030 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20240501 02:08:31.470206 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:31.470034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:60179)
0501 02:08:31.470124 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:31.470155 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":42,"threads_started":1}
I20240501 02:08:32.470840 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:32.470604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:48987)
0501 02:08:32.470723 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:32.470761 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20240501 02:08:32.470842 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:32.470562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:39041)
0501 02:08:32.470695 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:32.470792 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20240501 02:08:32.470870 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:32.470578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:34137)
0501 02:08:32.470803 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:32.470833 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":58,"threads_started":1}
I20240501 02:08:33.471472 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:33.471263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:56197)
0501 02:08:33.471373 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:33.471413 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20240501 02:08:33.471479 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:33.471284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:49979)
0501 02:08:33.471395 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:33.471426 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20240501 02:08:33.471495 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:33.471352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:44445)
0501 02:08:33.471433 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:33.471457 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":39,"threads_started":1}
I20240501 02:08:34.472054 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:34.471828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:32985)
0501 02:08:34.471946 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:34.471985 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20240501 02:08:34.472060 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:34.471815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:47395)
0501 02:08:34.471946 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:34.471985 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20240501 02:08:34.472083 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:34.471828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:44503)
0501 02:08:34.472017 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:34.472047 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":54,"threads_started":1}
I20240501 02:08:35.472718 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:35.472508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:36503)
0501 02:08:35.472624 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:35.472664 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":49,"threads_started":1}
I20240501 02:08:35.472713 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:35.472475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:47339)
0501 02:08:35.472618 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:35.472657 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20240501 02:08:35.472908 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:35.472475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:52141)
0501 02:08:35.472751 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:35.472781 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":71,"threads_started":1}
I20240501 02:08:36.473343 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:36.473129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:54869)
0501 02:08:36.473248 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:36.473286 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20240501 02:08:36.473369 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:36.473173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:41531)
0501 02:08:36.473290 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:36.473322 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20240501 02:08:36.473369 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:36.473208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:45615)
0501 02:08:36.473307 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:36.473330 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20240501 02:08:37.473930 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:37.473724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:35901)
0501 02:08:37.473843 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:37.473876 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20240501 02:08:37.473930 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:37.473724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:38355)
0501 02:08:37.473843 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:37.473876 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20240501 02:08:37.473956 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:37.473759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:55145)
0501 02:08:37.473885 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:37.473915 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20240501 02:08:38.474514 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:38.474351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:54287)
0501 02:08:38.474435 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:38.474458 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":44,"threads_started":1}
I20240501 02:08:38.474560 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:38.474330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:45209)
0501 02:08:38.474443 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:38.474496 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20240501 02:08:38.474514 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:38.474279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:36299)
0501 02:08:38.474423 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:38.474458 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20240501 02:08:39.475211 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:39.474908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:46101)
0501 02:08:39.475030 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:39.475071 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20240501 02:08:39.475225 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:39.474900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:58931)
0501 02:08:39.475139 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:39.475175 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":72,"threads_started":1}
I20240501 02:08:39.475353 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:39.475004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:35421)
0501 02:08:39.475276 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:39.475309 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":40,"threads_started":1}
I20240501 02:08:40.475932 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:40.475669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.67:38087)
0501 02:08:40.475797 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:40.475834 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20240501 02:08:40.476171 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:40.475770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.65:37735)
0501 02:08:40.476087 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:40.476120 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":46,"threads_started":1}
I20240501 02:08:40.476229 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:40.475670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35589 (local address 127.22.12.66:33797)
0501 02:08:40.476107 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:40.476145 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":70,"threads_started":1}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.22.12.126:35589,127.22.12.125:39095,127.22.12.124:38171, num_attempts: 1)
I20240501 02:08:40.628962 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 29450
I20240501 02:08:40.633807 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 29680
I20240501 02:08:40.638155 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 29816
I20240501 02:08:40.643764 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 29165
I20240501 02:08:40.648111 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 29356
2024-05-01T02:08:40Z chronyd exiting
I20240501 02:08:40.663470 22577 test_util.cc:175] -----------------------------------------------
I20240501 02:08:40.663538 22577 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714529249419714-22577-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:08:40 dist-test-slave-9tkh krb5kdc[30114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:08:40 dist-test-slave-9tkh krb5kdc[30114](info): set up 2 sockets
May 01 02:08:40 dist-test-slave-9tkh krb5kdc[30114](info): commencing operation
krb5kdc: starting...
W20240501 02:08:41.570520 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.906s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:08:41 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab.
Entry for principal oryx/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab.
I20240501 02:08:41.679343 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44723
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab
--principal=oryx/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:44723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:41.758390 30127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:41.758605 30127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:41.758634 30127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:41.759960 30127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:41.760005 30127 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240501 02:08:41.760021 30127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:41.760038 30127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:41.760051 30127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:41.760066 30127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:41.761679 30127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:44723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.126.keytab
--principal=oryx/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:41.761912 30127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:41.762091 30127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:41.766714 30135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:41.767089 30133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:41.767518 30132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:41.767930 30127 server_base.cc:1034] running on GCE node
I20240501 02:08:41.768152 30127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:41.768481 30127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:41.769637 30127 hybrid_clock.cc:648] HybridClock initialized: now 1714529321769616 us; error 34 us; skew 500 ppm
May 01 02:08:41 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529321, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:41.772086 30127 init.cc:377] Logged in from keytab as oryx/127.22.12.126@KRBTEST.COM (short username oryx)
I20240501 02:08:41.772529 30127 webserver.cc:460] Webserver started at http://127.22.12.126:38465/ using document root <none> and password file <none>
I20240501 02:08:41.772738 30127 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:41.772775 30127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:41.772873 30127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:41.773883 30127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "364665fbd08849639523ca1ad85b5462"
format_stamp: "Formatted at 2024-05-01 02:08:41 on dist-test-slave-9tkh"
server_key: "fb016888ac6075f5a40910b12cbd9f1e"
server_key_iv: "6ee1f9e5941c81f853cfa2adebf6b69a"
server_key_version: "encryptionkey@0"
I20240501 02:08:41.774152 30127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "364665fbd08849639523ca1ad85b5462"
format_stamp: "Formatted at 2024-05-01 02:08:41 on dist-test-slave-9tkh"
server_key: "fb016888ac6075f5a40910b12cbd9f1e"
server_key_iv: "6ee1f9e5941c81f853cfa2adebf6b69a"
server_key_version: "encryptionkey@0"
I20240501 02:08:41.775337 30127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:08:41.775987 30142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:41.776150 30127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:41.776225 30127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "364665fbd08849639523ca1ad85b5462"
format_stamp: "Formatted at 2024-05-01 02:08:41 on dist-test-slave-9tkh"
server_key: "fb016888ac6075f5a40910b12cbd9f1e"
server_key_iv: "6ee1f9e5941c81f853cfa2adebf6b69a"
server_key_version: "encryptionkey@0"
I20240501 02:08:41.776280 30127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:41.792001 30127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:41.794087 30127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:41.794190 30127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:41.798182 30127 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:44723
I20240501 02:08:41.798384 30204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:44723 every 8 connection(s)
I20240501 02:08:41.798550 30127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:41.799153 30205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:41.801211 30205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Bootstrap starting.
I20240501 02:08:41.801751 30205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:41.801980 30205 log.cc:824] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:41.802682 30205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: No bootstrap required, opened a new log
I20240501 02:08:41.803906 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 30127
I20240501 02:08:41.804010 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:41.804193 22577 external_mini_cluster.cc:1428] Setting key d12b42a2864a5fdf8e233a9b0697b534
I20240501 02:08:41.804296 30205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364665fbd08849639523ca1ad85b5462" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44723 } }
I20240501 02:08:41.804433 30205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:41.804457 30205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364665fbd08849639523ca1ad85b5462, State: Initialized, Role: FOLLOWER
I20240501 02:08:41.804622 30205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364665fbd08849639523ca1ad85b5462" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44723 } }
I20240501 02:08:41.804739 30205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:41.804801 30205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:41.804852 30205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:41.805791 30205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364665fbd08849639523ca1ad85b5462" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44723 } }
I20240501 02:08:41.805898 30205 leader_election.cc:302] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 364665fbd08849639523ca1ad85b5462; no voters:
I20240501 02:08:41.806054 30205 leader_election.cc:288] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:41.806106 30210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:41.806275 30205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:41.806280 30210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [term 1 LEADER]: Becoming Leader. State: Replica: 364665fbd08849639523ca1ad85b5462, State: Running, Role: LEADER
I20240501 02:08:41.806561 30210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364665fbd08849639523ca1ad85b5462" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44723 } }
I20240501 02:08:41.807013 30212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "364665fbd08849639523ca1ad85b5462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364665fbd08849639523ca1ad85b5462" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44723 } } }
I20240501 02:08:41.806998 30210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364665fbd08849639523ca1ad85b5462. Latest consensus state: current_term: 1 leader_uuid: "364665fbd08849639523ca1ad85b5462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364665fbd08849639523ca1ad85b5462" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44723 } } }
I20240501 02:08:41.807183 30212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:41.807188 30210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462 [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:41.807849 30219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:41.808458 30219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:41.809725 30219 catalog_manager.cc:1347] Generated new cluster ID: 7f44a61d47e743f8ab67fc9ee45893f3
I20240501 02:08:41.809767 30219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:08:41 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:41.814085 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:41.804641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.22.12.126:44723)
0501 02:08:41.804809 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:41.804824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:41.804848 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:08:41.804951 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:41.804956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:41.805006 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:41.805178 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:41.805830 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:41.807094 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:41.808045 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:41.808235 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:41.810655 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:41.810678 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:41.810679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:41.810697 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:41.812679 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:41.813042 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:41.813045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:41.813045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:41.813112 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:41.813254 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:41.813255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:41.813255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:41.813496 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:41.813643 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:41.813750 (+ 107us) server_negotiation.cc:299] Negotiation successful
0501 02:08:41.813885 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20240501 02:08:41.846294 30219 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:41.846922 30219 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:41.854074 30219 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Generated new TSK 0
I20240501 02:08:41.854274 30219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab.
Entry for principal oryx/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab.
I20240501 02:08:41.881095 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44723
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:41.982882 30231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:41.983070 30231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:41.983095 30231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:41.984431 30231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:41.984475 30231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:41.984494 30231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:08:41.984509 30231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:08:41.986177 30231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:44723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:41.986359 30231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:41.986548 30231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:41.991052 30237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:41.991320 30239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:41.991672 30236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:41.992084 30231 server_base.cc:1034] running on GCE node
I20240501 02:08:41.992255 30231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:41.992460 30231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:41.993589 30231 hybrid_clock.cc:648] HybridClock initialized: now 1714529321993577 us; error 30 us; skew 500 ppm
May 01 02:08:41 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529321, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:41.995942 30231 init.cc:377] Logged in from keytab as oryx/127.22.12.65@KRBTEST.COM (short username oryx)
I20240501 02:08:41.996371 30231 webserver.cc:460] Webserver started at http://127.22.12.65:45675/ using document root <none> and password file <none>
I20240501 02:08:41.996528 30231 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:41.996567 30231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:41.996670 30231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:41.997634 30231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "dbd56dd51583404b8a911564f7f35da9"
format_stamp: "Formatted at 2024-05-01 02:08:41 on dist-test-slave-9tkh"
server_key: "7fa0ec29b821b49d578a9ca839ca8e12"
server_key_iv: "9f34112926ea0598202426b70a237a20"
server_key_version: "encryptionkey@0"
I20240501 02:08:41.997918 30231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "dbd56dd51583404b8a911564f7f35da9"
format_stamp: "Formatted at 2024-05-01 02:08:41 on dist-test-slave-9tkh"
server_key: "7fa0ec29b821b49d578a9ca839ca8e12"
server_key_iv: "9f34112926ea0598202426b70a237a20"
server_key_version: "encryptionkey@0"
I20240501 02:08:41.999071 30231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:08:41.999682 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:41.999835 30231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:41.999907 30231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "dbd56dd51583404b8a911564f7f35da9"
format_stamp: "Formatted at 2024-05-01 02:08:41 on dist-test-slave-9tkh"
server_key: "7fa0ec29b821b49d578a9ca839ca8e12"
server_key_iv: "9f34112926ea0598202426b70a237a20"
server_key_version: "encryptionkey@0"
I20240501 02:08:41.999963 30231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:42.011370 30231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:42.013012 30231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:42.013123 30231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:42.013617 30231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:42.013654 30231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.013684 30231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:42.013707 30231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529321, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:42.020262 30231 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:45883
I20240501 02:08:42.020327 30361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:45883 every 8 connection(s)
I20240501 02:08:42.020632 30231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:08:42.021597 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.013980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:52987 (local address 127.22.12.126:44723)
0501 02:08:42.014046 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.014048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.014194 (+ 146us) server_negotiation.cc:407] Connection header received
0501 02:08:42.014718 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.014720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.014737 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.014820 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.015554 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.016134 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.016698 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.016864 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.018793 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.018798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.018798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.018806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.020447 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.020826 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.020827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.020827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.020855 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.021083 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.021084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.021085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.021195 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.021249 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.021449 (+ 200us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.021529 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:08:42.021677 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.013943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.22.12.65:52987)
0501 02:08:42.014078 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.014104 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.014610 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.014883 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.014888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.015156 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.015454 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.015459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.016224 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.016225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.016595 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.016596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.016727 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.017293 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.017309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.018701 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.020527 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.020530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.020533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.020749 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.020927 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.020927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.020928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.021013 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.021246 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.021248 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.021352 (+ 104us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.021432 (+ 80us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.021541 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20240501 02:08:42.022132 30362 heartbeater.cc:347] Connected to a master server at 127.22.12.126:44723
I20240501 02:08:42.022208 30362 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:42.022394 30362 heartbeater.cc:511] Master 127.22.12.126:44723 requested a full tablet report, sending...
I20240501 02:08:42.022927 30158 ts_manager.cc:194] Registered new tserver with Master: dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883)
I20240501 02:08:42.023926 30158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:52987
I20240501 02:08:42.026234 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 30231
I20240501 02:08:42.026329 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:08:42.026481 22577 external_mini_cluster.cc:1428] Setting key 558ac603920b9eb77da0b68213e0a438
I20240501 02:08:42.027809 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.023137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:36703 (local address 127.22.12.126:44723)
0501 02:08:42.023177 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.023179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.023266 (+ 87us) server_negotiation.cc:407] Connection header received
0501 02:08:42.023324 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.023325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.023332 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.023371 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.023881 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.024413 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.024988 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.025090 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.025689 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.025693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.025693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.025698 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.027008 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.027279 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.027281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.027281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.027304 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.027425 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.027426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.027426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.027505 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.027548 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.027686 (+ 138us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.027748 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:08:42.027885 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.023074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.22.12.65:36703)
0501 02:08:42.023188 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.023199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.023272 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.023384 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.023385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.023556 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.023809 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.023810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.024491 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.024492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.024926 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.024928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.024987 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.025400 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.025403 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.025624 (+ 221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.027076 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.027076 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.027077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.027205 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.027353 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.027353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.027354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.027383 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.027555 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.027555 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.027602 (+ 47us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.027668 (+ 66us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.027780 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab.
Entry for principal oryx/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab.
I20240501 02:08:42.049572 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44723
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:42.127969 30371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:42.128165 30371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:42.128190 30371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:42.129459 30371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:42.129508 30371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:42.129526 30371 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:08:42.129542 30371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:08:42.131168 30371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:44723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:42.131359 30371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:42.131533 30371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:42.135934 30379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:42.136162 30376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:42.136601 30377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:42.137042 30371 server_base.cc:1034] running on GCE node
I20240501 02:08:42.137239 30371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:42.137446 30371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:42.138576 30371 hybrid_clock.cc:648] HybridClock initialized: now 1714529322138559 us; error 32 us; skew 500 ppm
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:42.140916 30371 init.cc:377] Logged in from keytab as oryx/127.22.12.66@KRBTEST.COM (short username oryx)
I20240501 02:08:42.141343 30371 webserver.cc:460] Webserver started at http://127.22.12.66:43529/ using document root <none> and password file <none>
I20240501 02:08:42.141495 30371 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:42.141530 30371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:42.141610 30371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:42.142590 30371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "acfc992aecd1487f8d5dc32a90de815f"
format_stamp: "Formatted at 2024-05-01 02:08:42 on dist-test-slave-9tkh"
server_key: "ec43c3bcbacee2ba36df04823a1b1f85"
server_key_iv: "7bcde8fcff1bd774af142c922670ee76"
server_key_version: "encryptionkey@0"
I20240501 02:08:42.142851 30371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "acfc992aecd1487f8d5dc32a90de815f"
format_stamp: "Formatted at 2024-05-01 02:08:42 on dist-test-slave-9tkh"
server_key: "ec43c3bcbacee2ba36df04823a1b1f85"
server_key_iv: "7bcde8fcff1bd774af142c922670ee76"
server_key_version: "encryptionkey@0"
I20240501 02:08:42.144011 30371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:42.144629 30386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.144794 30371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:42.144867 30371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "acfc992aecd1487f8d5dc32a90de815f"
format_stamp: "Formatted at 2024-05-01 02:08:42 on dist-test-slave-9tkh"
server_key: "ec43c3bcbacee2ba36df04823a1b1f85"
server_key_iv: "7bcde8fcff1bd774af142c922670ee76"
server_key_version: "encryptionkey@0"
I20240501 02:08:42.144922 30371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:42.160100 30371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:42.162181 30371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:42.162292 30371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:42.162815 30371 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:42.162855 30371 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.162878 30371 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:42.162894 30371 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.168577 30371 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:46703
I20240501 02:08:42.168627 30501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:46703 every 8 connection(s)
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:42.168920 30371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:08:42.171372 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.163172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:39863 (local address 127.22.12.126:44723)
0501 02:08:42.163218 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.163220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.163431 (+ 211us) server_negotiation.cc:407] Connection header received
0501 02:08:42.164117 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.164118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.164132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.164203 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.165195 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.165953 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.166740 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.166924 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.169199 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.169210 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.169211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.169218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.170390 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.170765 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.170766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.170766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.170786 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.170911 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.170912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.170912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.170998 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.171041 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.171233 (+ 192us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.171305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:08:42.171466 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.163180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.22.12.66:39863)
0501 02:08:42.163326 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.163351 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.163980 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.164265 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.164270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.164673 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.165091 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.165097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.166046 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.166047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.166648 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.166651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.166795 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.167378 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.167393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.169107 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.170462 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.170466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.170468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.170696 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.170833 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.170833 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.170833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.170880 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.171035 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.171037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.171139 (+ 102us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.171217 (+ 78us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.171318 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20240501 02:08:42.171936 30502 heartbeater.cc:347] Connected to a master server at 127.22.12.126:44723
I20240501 02:08:42.172014 30502 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:42.172206 30502 heartbeater.cc:511] Master 127.22.12.126:44723 requested a full tablet report, sending...
I20240501 02:08:42.172570 30159 ts_manager.cc:194] Registered new tserver with Master: acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703)
I20240501 02:08:42.173053 30159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:39863
I20240501 02:08:42.173910 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 30371
I20240501 02:08:42.173995 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:08:42.174152 22577 external_mini_cluster.cc:1428] Setting key c669e99690e4c8901cf52ea8103135af
I20240501 02:08:42.177668 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.172916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.22.12.66:49729)
0501 02:08:42.173036 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.173046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.173105 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.173264 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.173265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.173392 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.173559 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.173561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.174263 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.174264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.174658 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.174659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.174717 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.175181 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.175185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.175416 (+ 231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.176922 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.176923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.176923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.177051 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.177182 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.177182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.177183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.177209 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.177430 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.177431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.177474 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.177510 (+ 36us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.177586 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20240501 02:08:42.177980 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.172982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:49729 (local address 127.22.12.126:44723)
0501 02:08:42.173022 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.173024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.173107 (+ 83us) server_negotiation.cc:407] Connection header received
0501 02:08:42.173181 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.173182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.173193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.173243 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.173643 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.174186 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.174722 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.175171 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.175488 (+ 317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.175490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.175491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.175496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.176852 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.177107 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.177108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.177108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.177128 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.177249 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.177249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.177250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.177354 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.177787 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.177834 (+ 47us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.177911 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab.
Entry for principal oryx/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab.
I20240501 02:08:42.197499 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44723
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:42.274901 30511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:42.275081 30511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:42.275113 30511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:42.276444 30511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:42.276491 30511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:42.276517 30511 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 02:08:42.276538 30511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:08:42.278182 30511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.12.126:44723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:42.278445 30511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:42.278676 30511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:42.283138 30516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:42.283553 30519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:42.283921 30517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:42.284345 30511 server_base.cc:1034] running on GCE node
I20240501 02:08:42.284520 30511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:42.284744 30511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:42.285876 30511 hybrid_clock.cc:648] HybridClock initialized: now 1714529322285863 us; error 29 us; skew 500 ppm
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:42.288268 30511 init.cc:377] Logged in from keytab as oryx/127.22.12.67@KRBTEST.COM (short username oryx)
I20240501 02:08:42.288687 30511 webserver.cc:460] Webserver started at http://127.22.12.67:41283/ using document root <none> and password file <none>
I20240501 02:08:42.288864 30511 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:42.288905 30511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:42.289008 30511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:42.289986 30511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "69bde8c8748144f3941647218a198e61"
format_stamp: "Formatted at 2024-05-01 02:08:42 on dist-test-slave-9tkh"
server_key: "b62729ee2977b12a84734b6fc3bdd62d"
server_key_iv: "8443dbb38633322d6f428a6b3a4fb582"
server_key_version: "encryptionkey@0"
I20240501 02:08:42.290328 30511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "69bde8c8748144f3941647218a198e61"
format_stamp: "Formatted at 2024-05-01 02:08:42 on dist-test-slave-9tkh"
server_key: "b62729ee2977b12a84734b6fc3bdd62d"
server_key_iv: "8443dbb38633322d6f428a6b3a4fb582"
server_key_version: "encryptionkey@0"
I20240501 02:08:42.291734 30511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:08:42.292398 30526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.292552 30511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:42.292630 30511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "69bde8c8748144f3941647218a198e61"
format_stamp: "Formatted at 2024-05-01 02:08:42 on dist-test-slave-9tkh"
server_key: "b62729ee2977b12a84734b6fc3bdd62d"
server_key_iv: "8443dbb38633322d6f428a6b3a4fb582"
server_key_version: "encryptionkey@0"
I20240501 02:08:42.292692 30511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:42.301014 30511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:42.303198 30511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:42.303313 30511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:42.303797 30511 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:42.303838 30511 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.303865 30511 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:42.303882 30511 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:42.309913 30511 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:41059
I20240501 02:08:42.309980 30641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:41059 every 8 connection(s)
I20240501 02:08:42.310281 30511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:08:42.311744 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 30511
I20240501 02:08:42.311851 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:08:42.312032 22577 external_mini_cluster.cc:1428] Setting key 9c0d03c4035d9b00ae596145e997fc07
I20240501 02:08:42.312965 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.304194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:55099 (local address 127.22.12.126:44723)
0501 02:08:42.304253 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.304256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.304423 (+ 167us) server_negotiation.cc:407] Connection header received
0501 02:08:42.304916 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.304918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.304932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.305012 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.306044 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.306861 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.307654 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.307833 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.310207 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.310212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.310212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.310221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.311574 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.312026 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.312027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.312027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.312052 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.312286 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.312287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.312287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.312419 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.312484 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.312788 (+ 304us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.312875 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:08:42.313066 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.304162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.22.12.67:55099)
0501 02:08:42.304312 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.304341 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.304814 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.305067 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.305072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.305466 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.305888 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.305895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.306953 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.306955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.307557 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.307560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.307713 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.308344 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.308372 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.310109 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.311652 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.311659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.311662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.311951 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.312111 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.312112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.312113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.312208 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.312484 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.312488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.312644 (+ 156us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.312765 (+ 121us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.312915 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20240501 02:08:42.313454 30642 heartbeater.cc:347] Connected to a master server at 127.22.12.126:44723
I20240501 02:08:42.313536 30642 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:42.313753 30642 heartbeater.cc:511] Master 127.22.12.126:44723 requested a full tablet report, sending...
I20240501 02:08:42.314131 30159 ts_manager.cc:194] Registered new tserver with Master: 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:42.314491 30159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:55099
I20240501 02:08:42.319100 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.314581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:58437 (local address 127.22.12.126:44723)
0501 02:08:42.314624 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.314626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.314894 (+ 268us) server_negotiation.cc:407] Connection header received
0501 02:08:42.314941 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.314942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.314949 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.314990 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.315465 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.316011 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.316485 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.316632 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.317178 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.317181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.317181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.317186 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.318398 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.318673 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.318674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.318674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.318692 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.318783 (+ 91us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.318783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.318784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.318856 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.318896 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.318984 (+ 88us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.319038 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:08:42.319118 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.314515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.22.12.67:58437)
0501 02:08:42.314823 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.314833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.314893 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.315002 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.315003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.315151 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.315373 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.315375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.316076 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.316077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.316417 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.316418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.316465 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.316894 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.316897 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.317106 (+ 209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.318461 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.318462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.318462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.318624 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.318731 (+ 107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.318732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.318732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.318753 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.318891 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.318892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.318924 (+ 32us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.318968 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.319036 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":85,"threads_started":1}
I20240501 02:08:42.323661 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 02:08:42.342396 30209 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:42.337660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.22.12.126:44723)
0501 02:08:42.337723 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.337725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.337735 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:08:42.337796 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.337797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.337805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.337852 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.338487 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.338995 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.339540 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.339722 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.342311 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36270: BlockingRecv error: recv got EOF from 127.0.0.1:36270 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:42.348800 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.343118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.22.12.126:44723)
0501 02:08:42.343184 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.343186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.343224 (+ 38us) server_negotiation.cc:407] Connection header received
0501 02:08:42.343277 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.343277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.343285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.343331 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.343853 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.344301 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.344797 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.344902 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.346713 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.346718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.346718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.346724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.348012 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.348308 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.348309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.348309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.348325 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.348425 (+ 100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.348425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.348425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.348511 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.348583 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.348654 (+ 71us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.348721 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 02:08:42.350407 30159 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:08:42.351827 30159 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:08:42.356812 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.353983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.0.0.1:50172)
0501 02:08:42.354298 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.354307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.354352 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.354664 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.354664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.354667 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.354882 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.354884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.355765 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.355766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.356538 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.356540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.356606 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.356646 (+ 40us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.356696 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.356733 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":34,"threads_started":1}
I20240501 02:08:42.357125 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.353785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41059 (local address 127.0.0.1:39250)
0501 02:08:42.353928 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.353954 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.354020 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.354403 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.354404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.354406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.354890 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.354891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.356207 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.356208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.356937 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.356940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.356998 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.357002 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.357026 (+ 24us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.357061 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20240501 02:08:42.357434 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.353786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:46703 (local address 127.0.0.1:57348)
0501 02:08:42.354046 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.354055 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.354119 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.354392 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.354396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.354397 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.354753 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.354756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.356120 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.356121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.357023 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.357026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.357075 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.357077 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.357103 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.357139 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20240501 02:08:42.357669 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.354315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.22.12.65:45883)
0501 02:08:42.354458 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.354465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.354473 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:42.354508 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.354509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.354522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.354648 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.354964 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.355681 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.356953 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.357466 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.357519 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.357553 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.357591 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20240501 02:08:42.357911 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.354073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.22.12.67:41059)
0501 02:08:42.354230 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.354236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.354244 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:42.354279 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.354280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.354296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.354926 (+ 630us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.355031 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.356135 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.357203 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.357711 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.357762 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.357796 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.357836 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20240501 02:08:42.358175 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.353882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.22.12.66:46703)
0501 02:08:42.354038 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.354046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.354116 (+ 70us) server_negotiation.cc:407] Connection header received
0501 02:08:42.354192 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.354194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.354211 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.354284 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.354841 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.356037 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.357240 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.358000 (+ 760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.358040 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.358071 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.358108 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20240501 02:08:42.358251 30296 tablet_service.cc:1467] Processing CreateTablet for tablet bc676a5198834a658b1296e7e6c2730b (DEFAULT_TABLE table=test-table [id=8585c67aaae24ee19a072d0a5b5d3b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.358521 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc676a5198834a658b1296e7e6c2730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.358741 30436 tablet_service.cc:1467] Processing CreateTablet for tablet bc676a5198834a658b1296e7e6c2730b (DEFAULT_TABLE table=test-table [id=8585c67aaae24ee19a072d0a5b5d3b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.358871 30576 tablet_service.cc:1467] Processing CreateTablet for tablet bc676a5198834a658b1296e7e6c2730b (DEFAULT_TABLE table=test-table [id=8585c67aaae24ee19a072d0a5b5d3b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.359017 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc676a5198834a658b1296e7e6c2730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.359114 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc676a5198834a658b1296e7e6c2730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.361174 30667 tablet_bootstrap.cc:492] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Bootstrap starting.
I20240501 02:08:42.361584 30668 tablet_bootstrap.cc:492] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Bootstrap starting.
I20240501 02:08:42.362160 30667 tablet_bootstrap.cc:654] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.362313 30668 tablet_bootstrap.cc:654] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.362418 30667 log.cc:824] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:42.362604 30668 log.cc:824] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:42.362666 30669 tablet_bootstrap.cc:492] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Bootstrap starting.
I20240501 02:08:42.363088 30667 tablet_bootstrap.cc:492] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: No bootstrap required, opened a new log
I20240501 02:08:42.363157 30667 ts_tablet_manager.cc:1367] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:08:42.363309 30668 tablet_bootstrap.cc:492] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: No bootstrap required, opened a new log
I20240501 02:08:42.363363 30668 ts_tablet_manager.cc:1367] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:08:42.363415 30669 tablet_bootstrap.cc:654] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.363636 30669 log.cc:824] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:42.364197 30669 tablet_bootstrap.cc:492] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: No bootstrap required, opened a new log
I20240501 02:08:42.364236 30669 ts_tablet_manager.cc:1367] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:08:42.364936 30667 raft_consensus.cc:348] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.364995 30668 raft_consensus.cc:348] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.365097 30668 raft_consensus.cc:374] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.365096 30667 raft_consensus.cc:374] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.365121 30668 raft_consensus.cc:729] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd56dd51583404b8a911564f7f35da9, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.365121 30667 raft_consensus.cc:729] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.365223 30667 consensus_queue.cc:260] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.365219 30668 consensus_queue.cc:260] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.365455 30667 ts_tablet_manager.cc:1398] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:08:42.365455 30668 ts_tablet_manager.cc:1398] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:08:42.365502 30362 heartbeater.cc:503] Master 127.22.12.126:44723 was elected leader, sending a full tablet report...
I20240501 02:08:42.365559 30502 heartbeater.cc:503] Master 127.22.12.126:44723 was elected leader, sending a full tablet report...
I20240501 02:08:42.366178 30669 raft_consensus.cc:348] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.366312 30669 raft_consensus.cc:374] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.366339 30669 raft_consensus.cc:729] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bde8c8748144f3941647218a198e61, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.366463 30669 consensus_queue.cc:260] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.366737 30669 ts_tablet_manager.cc:1398] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:08:42.366751 30642 heartbeater.cc:503] Master 127.22.12.126:44723 was elected leader, sending a full tablet report...
W20240501 02:08:42.419412 30503 tablet.cc:2369] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:08:42.511245 30675 raft_consensus.cc:482] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:42.511363 30675 raft_consensus.cc:504] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.511664 30675 leader_election.cc:288] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703), dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883)
I20240501 02:08:42.514196 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.511871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:46703 (local address 127.22.12.67:45499)
0501 02:08:42.511932 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.511944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.512008 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.512179 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.512179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.512182 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.512359 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.512361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.512991 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.512992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.513939 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.513940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.514014 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.514019 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.514050 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.514085 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240501 02:08:42.514214 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.511876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.22.12.67:35787)
0501 02:08:42.512056 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.512064 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.512119 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.512230 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.512231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.512233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.512440 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.512441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.513035 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.513035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.513929 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.513933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.514017 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.514020 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.514050 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.514085 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":12,"thread_start_us":118,"threads_started":1}
I20240501 02:08:42.514668 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.511946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:35787 (local address 127.22.12.65:45883)
0501 02:08:42.511990 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.511992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.512129 (+ 137us) server_negotiation.cc:407] Connection header received
0501 02:08:42.512167 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.512168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.512177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.512222 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.512508 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.512972 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.514031 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.514511 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.514520 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.514568 (+ 48us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.514600 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:08:42.514720 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.511916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:45499 (local address 127.22.12.66:46703)
0501 02:08:42.511953 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.511955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.511991 (+ 36us) server_negotiation.cc:407] Connection header received
0501 02:08:42.512093 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.512094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.512103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.512165 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.512428 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.512915 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.514031 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.514598 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.514606 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.514628 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.514663 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:08:42.515004 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc676a5198834a658b1296e7e6c2730b" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f" is_pre_election: true
I20240501 02:08:42.515005 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc676a5198834a658b1296e7e6c2730b" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9" is_pre_election: true
I20240501 02:08:42.515149 30456 raft_consensus.cc:2454] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 0.
I20240501 02:08:42.515156 30316 raft_consensus.cc:2454] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 0.
I20240501 02:08:42.515367 30529 leader_election.cc:302] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f; no voters:
I20240501 02:08:42.515534 30675 raft_consensus.cc:2781] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:08:42.515578 30675 raft_consensus.cc:482] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:08:42.515592 30675 raft_consensus.cc:3037] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.516216 30675 raft_consensus.cc:504] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.516417 30675 leader_election.cc:288] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 election: Requested vote from peers acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703), dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883)
I20240501 02:08:42.516510 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc676a5198834a658b1296e7e6c2730b" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f"
I20240501 02:08:42.516597 30456 raft_consensus.cc:3037] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.516577 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc676a5198834a658b1296e7e6c2730b" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9"
I20240501 02:08:42.516657 30316 raft_consensus.cc:3037] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.517272 30316 raft_consensus.cc:2454] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 1.
I20240501 02:08:42.517277 30456 raft_consensus.cc:2454] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 1.
I20240501 02:08:42.517448 30529 leader_election.cc:302] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f; no voters:
I20240501 02:08:42.517554 30675 raft_consensus.cc:2781] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:42.517743 30675 raft_consensus.cc:686] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 1 LEADER]: Becoming Leader. State: Replica: 69bde8c8748144f3941647218a198e61, State: Running, Role: LEADER
I20240501 02:08:42.517846 30675 consensus_queue.cc:237] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } }
I20240501 02:08:42.518679 30159 catalog_manager.cc:5484] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69bde8c8748144f3941647218a198e61 (127.22.12.67). New cstate: current_term: 1 leader_uuid: "69bde8c8748144f3941647218a198e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } health_report { overall_health: UNKNOWN } } }
W20240501 02:08:42.521274 30363 tablet.cc:2369] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:08:42.537930 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.535917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.22.12.67:41059)
0501 02:08:42.535980 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.535982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.535990 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:42.536028 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.536029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.536045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.536110 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:42.536360 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.536848 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.537419 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.537562 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.537599 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.537704 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.537771 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.537824 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.537853 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 02:08:42.539659 30316 raft_consensus.cc:1261] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Refusing update from remote peer 69bde8c8748144f3941647218a198e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:42.539659 30456 raft_consensus.cc:1261] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Refusing update from remote peer 69bde8c8748144f3941647218a198e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:42.540218 30677 consensus_queue.cc:1035] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:42.540341 30675 consensus_queue.cc:1035] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:42.544919 30159 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:42.545015 30159 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:42.545694 30159 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Sending DeleteTablet for 3 replicas of tablet bc676a5198834a658b1296e7e6c2730b
I20240501 02:08:42.546058 30576 tablet_service.cc:1514] Processing DeleteTablet for tablet bc676a5198834a658b1296e7e6c2730b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:42 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:39250
I20240501 02:08:42.546175 30436 tablet_service.cc:1514] Processing DeleteTablet for tablet bc676a5198834a658b1296e7e6c2730b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:42 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57348
I20240501 02:08:42.546312 30690 tablet_replica.cc:330] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: stopping tablet replica
I20240501 02:08:42.546394 30690 raft_consensus.cc:2229] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:08:42.546373 30296 tablet_service.cc:1514] Processing DeleteTablet for tablet bc676a5198834a658b1296e7e6c2730b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:42 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50172
I20240501 02:08:42.546381 30159 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 02:08:42.546494 30690 raft_consensus.cc:2258] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 02:08:42.546507 30159 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:08:42.546926 30690 ts_tablet_manager.cc:1875] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:42.546952 30691 tablet_replica.cc:330] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: stopping tablet replica
I20240501 02:08:42.547762 30692 tablet_replica.cc:330] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: stopping tablet replica
I20240501 02:08:42.547906 30692 raft_consensus.cc:2229] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:42.548023 30691 raft_consensus.cc:2229] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:42.548131 30296 tablet_service.cc:1467] Processing CreateTablet for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb (DEFAULT_TABLE table=test-table [id=99f31e654f144405bcad5e199f2823ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.548192 30692 raft_consensus.cc:2258] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:42.548254 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.548349 30691 raft_consensus.cc:2258] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:42.548796 30691 ts_tablet_manager.cc:1875] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:42.548803 30690 ts_tablet_manager.cc:1888] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:42.548863 30690 log.cc:1196] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/bc676a5198834a658b1296e7e6c2730b
I20240501 02:08:42.549134 30690 ts_tablet_manager.cc:1909] T bc676a5198834a658b1296e7e6c2730b P 69bde8c8748144f3941647218a198e61: Deleting consensus metadata
I20240501 02:08:42.549530 30668 tablet_bootstrap.cc:492] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Bootstrap starting.
I20240501 02:08:42.549552 30436 tablet_service.cc:1467] Processing CreateTablet for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb (DEFAULT_TABLE table=test-table [id=99f31e654f144405bcad5e199f2823ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.549618 30692 ts_tablet_manager.cc:1875] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:42.549641 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.549867 30668 tablet_bootstrap.cc:654] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.550402 30668 tablet_bootstrap.cc:492] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: No bootstrap required, opened a new log
I20240501 02:08:42.550439 30668 ts_tablet_manager.cc:1367] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:42.550580 30691 ts_tablet_manager.cc:1888] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:42.550635 30691 log.cc:1196] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/bc676a5198834a658b1296e7e6c2730b
I20240501 02:08:42.550623 30668 raft_consensus.cc:348] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.550689 30668 raft_consensus.cc:374] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.550709 30668 raft_consensus.cc:729] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd56dd51583404b8a911564f7f35da9, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.550750 30668 consensus_queue.cc:260] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.550802 30667 tablet_bootstrap.cc:492] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Bootstrap starting.
I20240501 02:08:42.550843 30668 ts_tablet_manager.cc:1398] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:42.551160 30667 tablet_bootstrap.cc:654] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.551461 30691 ts_tablet_manager.cc:1909] T bc676a5198834a658b1296e7e6c2730b P acfc992aecd1487f8d5dc32a90de815f: Deleting consensus metadata
I20240501 02:08:42.551766 30667 tablet_bootstrap.cc:492] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: No bootstrap required, opened a new log
I20240501 02:08:42.551815 30667 ts_tablet_manager.cc:1367] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:42.551887 30145 catalog_manager.cc:4830] TS acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703): tablet bc676a5198834a658b1296e7e6c2730b (table test-table [id=8585c67aaae24ee19a072d0a5b5d3b80]) successfully deleted
I20240501 02:08:42.551939 30667 raft_consensus.cc:348] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.551995 30667 raft_consensus.cc:374] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.551991 30144 catalog_manager.cc:4830] TS 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059): tablet bc676a5198834a658b1296e7e6c2730b (table test-table [id=8585c67aaae24ee19a072d0a5b5d3b80]) successfully deleted
I20240501 02:08:42.552013 30667 raft_consensus.cc:729] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.552054 30667 consensus_queue.cc:260] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.552062 30576 tablet_service.cc:1467] Processing CreateTablet for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb (DEFAULT_TABLE table=test-table [id=99f31e654f144405bcad5e199f2823ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.552139 30667 ts_tablet_manager.cc:1398] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.552152 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.553128 30669 tablet_bootstrap.cc:492] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Bootstrap starting.
I20240501 02:08:42.553499 30669 tablet_bootstrap.cc:654] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.554111 30669 tablet_bootstrap.cc:492] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: No bootstrap required, opened a new log
I20240501 02:08:42.554163 30669 ts_tablet_manager.cc:1367] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:42.554311 30669 raft_consensus.cc:348] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.554378 30669 raft_consensus.cc:374] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.554396 30669 raft_consensus.cc:729] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bde8c8748144f3941647218a198e61, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.554448 30669 consensus_queue.cc:260] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.554586 30669 ts_tablet_manager.cc:1398] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:42.555011 30692 ts_tablet_manager.cc:1888] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:42.555088 30692 log.cc:1196] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/bc676a5198834a658b1296e7e6c2730b
I20240501 02:08:42.555286 30692 ts_tablet_manager.cc:1909] T bc676a5198834a658b1296e7e6c2730b P dbd56dd51583404b8a911564f7f35da9: Deleting consensus metadata
I20240501 02:08:42.555845 30144 catalog_manager.cc:4830] TS dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883): tablet bc676a5198834a658b1296e7e6c2730b (table test-table [id=8585c67aaae24ee19a072d0a5b5d3b80]) successfully deleted
W20240501 02:08:42.560848 30643 tablet.cc:2369] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:08:42.726912 30686 raft_consensus.cc:482] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:42.727018 30686 raft_consensus.cc:504] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.727159 30686 leader_election.cc:288] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883), acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703)
I20240501 02:08:42.727408 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523712cf2cdc4da78c6f90bbb9ee0dbb" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f" is_pre_election: true
I20240501 02:08:42.727387 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523712cf2cdc4da78c6f90bbb9ee0dbb" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9" is_pre_election: true
I20240501 02:08:42.727478 30674 raft_consensus.cc:482] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:42.727501 30456 raft_consensus.cc:2454] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 0.
I20240501 02:08:42.727522 30674 raft_consensus.cc:504] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.727640 30316 raft_consensus.cc:2454] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 0.
I20240501 02:08:42.727763 30529 leader_election.cc:302] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f; no voters:
I20240501 02:08:42.727809 30674 leader_election.cc:288] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:42.727838 30686 raft_consensus.cc:2781] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:08:42.727871 30686 raft_consensus.cc:482] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:08:42.727891 30686 raft_consensus.cc:3037] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.728845 30686 raft_consensus.cc:504] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.728987 30686 leader_election.cc:288] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 election: Requested vote from peers dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883), acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703)
I20240501 02:08:42.729212 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523712cf2cdc4da78c6f90bbb9ee0dbb" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9"
I20240501 02:08:42.729293 30316 raft_consensus.cc:3037] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.729563 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523712cf2cdc4da78c6f90bbb9ee0dbb" candidate_uuid: "69bde8c8748144f3941647218a198e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f"
I20240501 02:08:42.729645 30456 raft_consensus.cc:3037] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.730038 30316 raft_consensus.cc:2454] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 1.
I20240501 02:08:42.730222 30528 leader_election.cc:302] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, dbd56dd51583404b8a911564f7f35da9; no voters:
I20240501 02:08:42.730306 30686 raft_consensus.cc:2781] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:42.730336 30686 raft_consensus.cc:686] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 1 LEADER]: Becoming Leader. State: Replica: 69bde8c8748144f3941647218a198e61, State: Running, Role: LEADER
I20240501 02:08:42.730379 30686 consensus_queue.cc:237] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.730399 30456 raft_consensus.cc:2454] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69bde8c8748144f3941647218a198e61 in term 1.
I20240501 02:08:42.731029 30158 catalog_manager.cc:5484] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69bde8c8748144f3941647218a198e61 (127.22.12.67). New cstate: current_term: 1 leader_uuid: "69bde8c8748144f3941647218a198e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } health_report { overall_health: HEALTHY } } }
I20240501 02:08:42.731199 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.728081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:33667 (local address 127.22.12.66:46703)
0501 02:08:42.728167 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.728170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.728177 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:08:42.728227 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.728227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.728237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.728292 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.728828 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.729490 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.730562 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.730986 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.730994 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.731105 (+ 111us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.731143 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20240501 02:08:42.731216 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.728083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:57821 (local address 127.22.12.67:41059)
0501 02:08:42.728137 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.728139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.728227 (+ 88us) server_negotiation.cc:407] Connection header received
0501 02:08:42.728314 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.728315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.728325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.728351 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:42.728732 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.729437 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.730497 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.730981 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.730991 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.731114 (+ 123us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.731158 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:08:42.731489 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.727948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41059 (local address 127.22.12.65:57821)
0501 02:08:42.728146 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.728157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.728253 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.728410 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.728411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.728412 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.728645 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.728648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.729506 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.729507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.730374 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.730376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.731025 (+ 649us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.731029 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.731305 (+ 276us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.731348 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":12,"thread_start_us":72,"threads_started":1}
I20240501 02:08:42.731489 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.727948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:46703 (local address 127.22.12.65:33667)
0501 02:08:42.728103 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.728114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.728171 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.728298 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.728298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.728300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:42.728731 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.728734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.729580 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.729581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.730445 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.730447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.731022 (+ 575us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.731026 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.731273 (+ 247us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.731323 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
I20240501 02:08:42.731738 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523712cf2cdc4da78c6f90bbb9ee0dbb" candidate_uuid: "dbd56dd51583404b8a911564f7f35da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61" is_pre_election: true
I20240501 02:08:42.731756 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "523712cf2cdc4da78c6f90bbb9ee0dbb" candidate_uuid: "dbd56dd51583404b8a911564f7f35da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f" is_pre_election: true
I20240501 02:08:42.731838 30456 raft_consensus.cc:2379] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbd56dd51583404b8a911564f7f35da9 in current term 1: Already voted for candidate 69bde8c8748144f3941647218a198e61 in this term.
I20240501 02:08:42.732025 30248 leader_election.cc:302] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbd56dd51583404b8a911564f7f35da9; no voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f
I20240501 02:08:42.732168 30674 raft_consensus.cc:2726] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529321, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.126@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:42.801765 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.795359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44723 (local address 127.0.0.1:36274)
0501 02:08:42.795498 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.795507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.795581 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.795721 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.795721 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.796112 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:42.796305 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.796308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.796924 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.796925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.797285 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.797287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.797349 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.797878 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:42.797882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:42.799433 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:42.801028 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.801033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.801036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.801218 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.801339 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:42.801339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:42.801340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:42.801362 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:42.801522 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:42.801525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:42.801565 (+ 40us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.801612 (+ 47us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.801699 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20240501 02:08:42.801842 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.795386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.22.12.126:44723)
0501 02:08:42.795435 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.795436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.795571 (+ 135us) server_negotiation.cc:407] Connection header received
0501 02:08:42.795638 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.795638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.795666 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.795712 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.796385 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.796852 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.797374 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.797523 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.799552 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.799557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.799558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.799564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.800954 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.801277 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.801278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.801278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.801294 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.801401 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.801401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.801402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.801483 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.801525 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.801633 (+ 108us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.801750 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":28}
I20240501 02:08:42.803857 30158 catalog_manager.cc:1904] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:36274:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240501 02:08:42.804078 30158 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:08:42.805428 30576 tablet_service.cc:1467] Processing CreateTablet for tablet 70bcecfcffff4c009e8990bbe5ede06e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89c95bfc4ad14e698407951c4751d116]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:08:42.805423 30436 tablet_service.cc:1467] Processing CreateTablet for tablet 70bcecfcffff4c009e8990bbe5ede06e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89c95bfc4ad14e698407951c4751d116]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:08:42.805423 30296 tablet_service.cc:1467] Processing CreateTablet for tablet 70bcecfcffff4c009e8990bbe5ede06e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89c95bfc4ad14e698407951c4751d116]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 02:08:42.805568 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70bcecfcffff4c009e8990bbe5ede06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.805568 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70bcecfcffff4c009e8990bbe5ede06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.805569 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70bcecfcffff4c009e8990bbe5ede06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.806809 30669 tablet_bootstrap.cc:492] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: Bootstrap starting.
I20240501 02:08:42.806985 30667 tablet_bootstrap.cc:492] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: Bootstrap starting.
I20240501 02:08:42.807126 30669 tablet_bootstrap.cc:654] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.807404 30667 tablet_bootstrap.cc:654] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.807740 30668 tablet_bootstrap.cc:492] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: Bootstrap starting.
I20240501 02:08:42.807866 30669 tablet_bootstrap.cc:492] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: No bootstrap required, opened a new log
I20240501 02:08:42.807920 30669 ts_tablet_manager.cc:1367] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:42.808019 30667 tablet_bootstrap.cc:492] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: No bootstrap required, opened a new log
I20240501 02:08:42.808059 30667 ts_tablet_manager.cc:1367] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:42.808043 30669 raft_consensus.cc:348] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.808094 30669 raft_consensus.cc:374] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.808106 30669 raft_consensus.cc:729] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bde8c8748144f3941647218a198e61, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.808107 30668 tablet_bootstrap.cc:654] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.808166 30669 consensus_queue.cc:260] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.808188 30667 raft_consensus.cc:348] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.808257 30669 ts_tablet_manager.cc:1398] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.808265 30667 raft_consensus.cc:374] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.808284 30667 raft_consensus.cc:729] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.808279 30686 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.808329 30667 consensus_queue.cc:260] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.808338 30686 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.808423 30667 ts_tablet_manager.cc:1398] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:42.808552 30673 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.808630 30673 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.808727 30668 tablet_bootstrap.cc:492] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: No bootstrap required, opened a new log
I20240501 02:08:42.808780 30668 ts_tablet_manager.cc:1367] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:42.808903 30668 raft_consensus.cc:348] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.808964 30668 raft_consensus.cc:374] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.808975 30668 raft_consensus.cc:729] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd56dd51583404b8a911564f7f35da9, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.809006 30668 consensus_queue.cc:260] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.809074 30668 ts_tablet_manager.cc:1398] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.809185 30674 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.809255 30674 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.885057 30674 raft_consensus.cc:482] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:42.885160 30674 raft_consensus.cc:504] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.885318 30674 leader_election.cc:288] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:42.885509 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70bcecfcffff4c009e8990bbe5ede06e" candidate_uuid: "dbd56dd51583404b8a911564f7f35da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61" is_pre_election: true
I20240501 02:08:42.885532 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70bcecfcffff4c009e8990bbe5ede06e" candidate_uuid: "dbd56dd51583404b8a911564f7f35da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f" is_pre_election: true
I20240501 02:08:42.885602 30456 raft_consensus.cc:2454] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd56dd51583404b8a911564f7f35da9 in term 0.
I20240501 02:08:42.885612 30596 raft_consensus.cc:2454] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd56dd51583404b8a911564f7f35da9 in term 0.
I20240501 02:08:42.885759 30249 leader_election.cc:302] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acfc992aecd1487f8d5dc32a90de815f, dbd56dd51583404b8a911564f7f35da9; no voters:
I20240501 02:08:42.885880 30674 raft_consensus.cc:2781] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:08:42.885934 30674 raft_consensus.cc:482] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:08:42.885952 30674 raft_consensus.cc:3037] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.886641 30674 raft_consensus.cc:504] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.886777 30674 leader_election.cc:288] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [CANDIDATE]: Term 1 election: Requested vote from peers acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:42.886945 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70bcecfcffff4c009e8990bbe5ede06e" candidate_uuid: "dbd56dd51583404b8a911564f7f35da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61"
I20240501 02:08:42.886952 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70bcecfcffff4c009e8990bbe5ede06e" candidate_uuid: "dbd56dd51583404b8a911564f7f35da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfc992aecd1487f8d5dc32a90de815f"
I20240501 02:08:42.887012 30596 raft_consensus.cc:3037] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.887012 30456 raft_consensus.cc:3037] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:42.887557 30456 raft_consensus.cc:2454] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd56dd51583404b8a911564f7f35da9 in term 1.
I20240501 02:08:42.887682 30596 raft_consensus.cc:2454] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd56dd51583404b8a911564f7f35da9 in term 1.
I20240501 02:08:42.887753 30249 leader_election.cc:302] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acfc992aecd1487f8d5dc32a90de815f, dbd56dd51583404b8a911564f7f35da9; no voters:
I20240501 02:08:42.887840 30674 raft_consensus.cc:2781] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:42.887995 30674 raft_consensus.cc:686] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [term 1 LEADER]: Becoming Leader. State: Replica: dbd56dd51583404b8a911564f7f35da9, State: Running, Role: LEADER
I20240501 02:08:42.888063 30674 consensus_queue.cc:237] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.888401 30708 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: TxnStatusTablet state changed. Reason: New leader dbd56dd51583404b8a911564f7f35da9. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.888532 30708 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:08:42.888744 30710 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240501 02:08:42.888846 30710 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240501 02:08:42.888777 30158 catalog_manager.cc:5484] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbd56dd51583404b8a911564f7f35da9 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20240501 02:08:42.902726 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.900283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.0.0.1:50188)
0501 02:08:42.900351 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.900364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.900414 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.900572 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.900573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.900574 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:08:42.900734 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.900735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.901615 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.901616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.902043 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.902045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.902094 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.902109 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.902562 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.902572 (+ 10us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.902616 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.902661 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 02:08:42.902741 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.900366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.22.12.65:45883)
0501 02:08:42.900430 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.900433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.900443 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:08:42.900479 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.900480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.900499 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.900573 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:42.900794 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.901543 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.902103 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.902292 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.902331 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.902463 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.902561 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.902635 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.902661 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:08:42.903887 30456 raft_consensus.cc:1261] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Refusing update from remote peer dbd56dd51583404b8a911564f7f35da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:42.903931 30596 raft_consensus.cc:1261] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Refusing update from remote peer dbd56dd51583404b8a911564f7f35da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:42.904155 30674 consensus_queue.cc:1035] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:42.904263 30708 consensus_queue.cc:1035] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:42.904830 30673 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: TxnStatusTablet state changed. Reason: New leader dbd56dd51583404b8a911564f7f35da9. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.904844 30686 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: TxnStatusTablet state changed. Reason: New leader dbd56dd51583404b8a911564f7f35da9. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.904913 30673 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.904919 30686 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.905673 30680 mvcc.cc:205] Tried to move back new op lower bound from 7022712106612465664 to 7022712106549981184. Current Snapshot: MvccSnapshot[applied={T|T < 7022712106612465664}]
I20240501 02:08:42.905692 30674 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.905772 30674 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:08:42.905754 30686 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.905808 30686 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P 69bde8c8748144f3941647218a198e61: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.906105 30674 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.906167 30674 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P dbd56dd51583404b8a911564f7f35da9: This TxnStatusTablet replica's current role is: LEADER
I20240501 02:08:42.906244 30682 mvcc.cc:205] Tried to move back new op lower bound from 7022712106612465664 to 7022712106549981184. Current Snapshot: MvccSnapshot[applied={T|T < 7022712106612465664}]
I20240501 02:08:42.906328 30673 tablet_replica.cc:439] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbd56dd51583404b8a911564f7f35da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } } }
I20240501 02:08:42.906383 30673 tablet_replica.cc:442] T 70bcecfcffff4c009e8990bbe5ede06e P acfc992aecd1487f8d5dc32a90de815f: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 02:08:42.911226 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.909013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.22.12.67:54653)
0501 02:08:42.909162 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.909173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.909232 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.909344 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.909345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.909346 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:08:42.909500 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.909501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.910328 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.910328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.910752 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.910753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.910805 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.910817 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.911085 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.911086 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.911120 (+ 34us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.911150 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20240501 02:08:42.911235 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.909086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:54653 (local address 127.22.12.65:45883)
0501 02:08:42.909127 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.909135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.909229 (+ 94us) server_negotiation.cc:407] Connection header received
0501 02:08:42.909271 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.909272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.909281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.909339 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:42.909551 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.910265 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.910813 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.910972 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.910991 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.911019 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.911081 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.911146 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.911168 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:08:42.913231 30316 raft_consensus.cc:1261] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Refusing update from remote peer 69bde8c8748144f3941647218a198e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:42.913283 30456 raft_consensus.cc:1261] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Refusing update from remote peer 69bde8c8748144f3941647218a198e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:42.913456 30675 consensus_queue.cc:1035] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:42.913558 30686 consensus_queue.cc:1035] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:42.920836 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.918638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:37075 (local address 127.22.12.67:41059)
0501 02:08:42.918676 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.918678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.918858 (+ 180us) server_negotiation.cc:407] Connection header received
0501 02:08:42.918908 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.918909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.918919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.918967 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:42.919249 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.919685 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.920370 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.920527 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.920548 (+ 21us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.920578 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.920622 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.920762 (+ 140us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.920783 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:08:42.920836 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.918614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41059 (local address 127.22.12.65:37075)
0501 02:08:42.918786 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:42.918797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:42.918855 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:42.918972 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:42.918973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:42.918974 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:08:42.919190 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.919192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.919770 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.919771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:42.920211 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:42.920213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.920288 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.920308 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:08:42.920646 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:08:42.920648 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:08:42.920688 (+ 40us) client_negotiation.cc:240] Negotiation successful
0501 02:08:42.920735 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20240501 02:08:42.923718 30681 mvcc.cc:205] Tried to move back new op lower bound from 7022712106651033600 to 7022712105903910912. Current Snapshot: MvccSnapshot[applied={T|T < 7022712106651033600}]
I20240501 02:08:42.929837 30157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:42.929951 30157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:42.930588 30157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Sending DeleteTablet for 3 replicas of tablet 523712cf2cdc4da78c6f90bbb9ee0dbb
I20240501 02:08:42.930826 30436 tablet_service.cc:1514] Processing DeleteTablet for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:42 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57348
I20240501 02:08:42.930862 30296 tablet_service.cc:1514] Processing DeleteTablet for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:42 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50172
I20240501 02:08:42.930928 30691 tablet_replica.cc:330] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: stopping tablet replica
I20240501 02:08:42.930967 30692 tablet_replica.cc:330] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: stopping tablet replica
I20240501 02:08:42.930991 30691 raft_consensus.cc:2229] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:42.931030 30691 raft_consensus.cc:2258] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:42.931028 30692 raft_consensus.cc:2229] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:42.931067 30692 raft_consensus.cc:2258] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:42.931427 30691 ts_tablet_manager.cc:1875] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:42.932857 30691 ts_tablet_manager.cc:1888] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:08:42.932911 30691 log.cc:1196] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/523712cf2cdc4da78c6f90bbb9ee0dbb
I20240501 02:08:42.933101 30691 ts_tablet_manager.cc:1909] T 523712cf2cdc4da78c6f90bbb9ee0dbb P acfc992aecd1487f8d5dc32a90de815f: Deleting consensus metadata
I20240501 02:08:42.933391 30145 catalog_manager.cc:4830] TS acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703): tablet 523712cf2cdc4da78c6f90bbb9ee0dbb (table test-table [id=99f31e654f144405bcad5e199f2823ea]) successfully deleted
I20240501 02:08:42.934320 30576 tablet_service.cc:1514] Processing DeleteTablet for tablet 523712cf2cdc4da78c6f90bbb9ee0dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:42 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:39250
I20240501 02:08:42.934482 30692 ts_tablet_manager.cc:1875] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:42.934597 30690 tablet_replica.cc:330] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: stopping tablet replica
I20240501 02:08:42.934667 30690 raft_consensus.cc:2229] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:08:42.934737 30690 raft_consensus.cc:2258] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:42.934973 30690 ts_tablet_manager.cc:1875] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:42.935981 30692 ts_tablet_manager.cc:1888] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:08:42.936036 30692 log.cc:1196] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/523712cf2cdc4da78c6f90bbb9ee0dbb
I20240501 02:08:42.936200 30692 ts_tablet_manager.cc:1909] T 523712cf2cdc4da78c6f90bbb9ee0dbb P dbd56dd51583404b8a911564f7f35da9: Deleting consensus metadata
I20240501 02:08:42.936399 30690 ts_tablet_manager.cc:1888] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:08:42.936455 30690 log.cc:1196] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/523712cf2cdc4da78c6f90bbb9ee0dbb
I20240501 02:08:42.936489 30144 catalog_manager.cc:4830] TS dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883): tablet 523712cf2cdc4da78c6f90bbb9ee0dbb (table test-table [id=99f31e654f144405bcad5e199f2823ea]) successfully deleted
I20240501 02:08:42.936635 30690 ts_tablet_manager.cc:1909] T 523712cf2cdc4da78c6f90bbb9ee0dbb P 69bde8c8748144f3941647218a198e61: Deleting consensus metadata
I20240501 02:08:42.936902 30144 catalog_manager.cc:4830] TS 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059): tablet 523712cf2cdc4da78c6f90bbb9ee0dbb (table test-table [id=99f31e654f144405bcad5e199f2823ea]) successfully deleted
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 02:08:42.954273 30209 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:42.949396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.22.12.126:44723)
0501 02:08:42.949451 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.949454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.949577 (+ 123us) server_negotiation.cc:407] Connection header received
0501 02:08:42.949665 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.949666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.949675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.949723 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.950379 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.950914 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.951469 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.951626 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.954200 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36282: BlockingRecv error: recv got EOF from 127.0.0.1:36282 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
May 01 02:08:42 dist-test-slave-9tkh krb5kdc[30114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM
I20240501 02:08:42.961098 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:42.955163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.22.12.126:44723)
0501 02:08:42.955207 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:42.955209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:42.955225 (+ 16us) server_negotiation.cc:407] Connection header received
0501 02:08:42.955267 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:42.955268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:42.955277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:42.955325 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:42.955844 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.956302 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:42.956786 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:42.956935 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:42.958982 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:42.958986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:42.958986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:42.958993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:42.960250 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.960510 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.960511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.960512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.960532 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:42.960657 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:42.960657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:42.960658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:42.960746 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:42.960835 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:42.960928 (+ 93us) server_negotiation.cc:299] Negotiation successful
0501 02:08:42.961009 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:08:42.962693 30157 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:08:42.962859 30157 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:08:42.964174 30576 tablet_service.cc:1467] Processing CreateTablet for tablet e78b2e3b60c740b8993f0699bc3ade32 (DEFAULT_TABLE table=test-table [id=79d6dcdba45247d882525134dee78155]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.964200 30436 tablet_service.cc:1467] Processing CreateTablet for tablet e78b2e3b60c740b8993f0699bc3ade32 (DEFAULT_TABLE table=test-table [id=79d6dcdba45247d882525134dee78155]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.964205 30296 tablet_service.cc:1467] Processing CreateTablet for tablet e78b2e3b60c740b8993f0699bc3ade32 (DEFAULT_TABLE table=test-table [id=79d6dcdba45247d882525134dee78155]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:42.964310 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78b2e3b60c740b8993f0699bc3ade32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.964311 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78b2e3b60c740b8993f0699bc3ade32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.964329 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78b2e3b60c740b8993f0699bc3ade32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:42.965533 30669 tablet_bootstrap.cc:492] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Bootstrap starting.
I20240501 02:08:42.965622 30668 tablet_bootstrap.cc:492] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Bootstrap starting.
I20240501 02:08:42.965657 30667 tablet_bootstrap.cc:492] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Bootstrap starting.
I20240501 02:08:42.965844 30669 tablet_bootstrap.cc:654] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.966084 30667 tablet_bootstrap.cc:654] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.966154 30668 tablet_bootstrap.cc:654] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:42.966840 30667 tablet_bootstrap.cc:492] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: No bootstrap required, opened a new log
I20240501 02:08:42.966894 30667 ts_tablet_manager.cc:1367] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:42.967028 30669 tablet_bootstrap.cc:492] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: No bootstrap required, opened a new log
I20240501 02:08:42.967034 30667 raft_consensus.cc:348] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.967084 30667 raft_consensus.cc:374] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.967083 30669 ts_tablet_manager.cc:1367] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:08:42.967097 30667 raft_consensus.cc:729] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.967133 30667 consensus_queue.cc:260] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.967186 30669 raft_consensus.cc:348] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.967223 30667 ts_tablet_manager.cc:1398] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:42.967244 30669 raft_consensus.cc:374] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.967263 30669 raft_consensus.cc:729] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bde8c8748144f3941647218a198e61, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.967305 30669 consensus_queue.cc:260] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.967386 30669 ts_tablet_manager.cc:1398] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:42.967657 30668 tablet_bootstrap.cc:492] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: No bootstrap required, opened a new log
I20240501 02:08:42.967715 30668 ts_tablet_manager.cc:1367] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:08:42.967860 30668 raft_consensus.cc:348] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.967922 30668 raft_consensus.cc:374] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:42.967938 30668 raft_consensus.cc:729] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd56dd51583404b8a911564f7f35da9, State: Initialized, Role: FOLLOWER
I20240501 02:08:42.967974 30668 consensus_queue.cc:260] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:42.968060 30668 ts_tablet_manager.cc:1398] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:43.053844 30673 raft_consensus.cc:482] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:43.053973 30673 raft_consensus.cc:504] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.054366 30673 leader_election.cc:288] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:43.057591 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.054842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41059 (local address 127.22.12.66:54657)
0501 02:08:43.054960 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:43.054970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:43.055080 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:43.055199 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:43.055199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:43.055201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:43.055395 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.055397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.056254 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.056254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:43.057249 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.057251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.057331 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.057334 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:08:43.057365 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:08:43.057401 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20240501 02:08:43.057631 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.054638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.22.12.66:60029)
0501 02:08:43.054798 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:43.054811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:43.054892 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:43.055046 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:43.055047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:43.055050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:08:43.055305 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.055308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.056254 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.056254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:43.057249 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.057251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.057330 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.057334 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:08:43.057365 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:08:43.057401 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20240501 02:08:43.057907 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.054696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:60029 (local address 127.22.12.65:45883)
0501 02:08:43.054784 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:43.054786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:43.054872 (+ 86us) server_negotiation.cc:407] Connection header received
0501 02:08:43.054945 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:43.054946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:43.054955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:43.055011 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:43.055405 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.056156 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.057340 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.057795 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.057801 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:43.057819 (+ 18us) server_negotiation.cc:299] Negotiation successful
0501 02:08:43.057843 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 02:08:43.057907 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.054936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:54657 (local address 127.22.12.67:41059)
0501 02:08:43.054986 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:43.054988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:43.055026 (+ 38us) server_negotiation.cc:407] Connection header received
0501 02:08:43.055147 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:43.055148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:43.055155 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:43.055193 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:08:43.055447 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.056168 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.057340 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.057784 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.057793 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:43.057815 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:08:43.057843 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:08:43.058101 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78b2e3b60c740b8993f0699bc3ade32" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61" is_pre_election: true
I20240501 02:08:43.058137 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78b2e3b60c740b8993f0699bc3ade32" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9" is_pre_election: true
I20240501 02:08:43.058192 30596 raft_consensus.cc:2454] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 0.
I20240501 02:08:43.058213 30316 raft_consensus.cc:2454] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 0.
I20240501 02:08:43.058380 30388 leader_election.cc:302] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f; no voters:
I20240501 02:08:43.058559 30673 raft_consensus.cc:2781] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:08:43.058631 30673 raft_consensus.cc:482] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:08:43.058652 30673 raft_consensus.cc:3037] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:43.059278 30673 raft_consensus.cc:504] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.059399 30673 leader_election.cc:288] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 election: Requested vote from peers dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:43.059556 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78b2e3b60c740b8993f0699bc3ade32" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9"
I20240501 02:08:43.059640 30316 raft_consensus.cc:3037] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:43.059612 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78b2e3b60c740b8993f0699bc3ade32" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61"
I20240501 02:08:43.059700 30596 raft_consensus.cc:3037] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:43.060374 30596 raft_consensus.cc:2454] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 1.
I20240501 02:08:43.060374 30316 raft_consensus.cc:2454] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 1.
I20240501 02:08:43.060547 30388 leader_election.cc:302] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acfc992aecd1487f8d5dc32a90de815f, dbd56dd51583404b8a911564f7f35da9; no voters:
I20240501 02:08:43.060662 30673 raft_consensus.cc:2781] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:43.060812 30673 raft_consensus.cc:686] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 1 LEADER]: Becoming Leader. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Running, Role: LEADER
I20240501 02:08:43.060880 30673 consensus_queue.cc:237] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.061564 30158 catalog_manager.cc:5484] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f reported cstate change: term changed from 0 to 1, leader changed from <none> to acfc992aecd1487f8d5dc32a90de815f (127.22.12.66). New cstate: current_term: 1 leader_uuid: "acfc992aecd1487f8d5dc32a90de815f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20240501 02:08:43.084779 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.082964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.22.12.66:46703)
0501 02:08:43.083017 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:43.083019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:43.083027 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:43.083072 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:43.083073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:43.083090 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:43.083157 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:43.083372 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.083834 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.084359 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.084460 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.084496 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:43.084587 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:43.084644 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:43.084674 (+ 30us) server_negotiation.cc:299] Negotiation successful
0501 02:08:43.084694 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:08:43.085762 30596 raft_consensus.cc:1261] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Refusing update from remote peer acfc992aecd1487f8d5dc32a90de815f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:43.085783 30316 raft_consensus.cc:1261] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Refusing update from remote peer acfc992aecd1487f8d5dc32a90de815f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:43.086041 30743 consensus_queue.cc:1035] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:43.086159 30673 consensus_queue.cc:1035] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [LEADER]: Connected to new peer: Peer: permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:43.089460 30158 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:43.089546 30158 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:43.090234 30158 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Sending DeleteTablet for 3 replicas of tablet e78b2e3b60c740b8993f0699bc3ade32
I20240501 02:08:43.090553 30576 tablet_service.cc:1514] Processing DeleteTablet for tablet e78b2e3b60c740b8993f0699bc3ade32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:43 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:39250
I20240501 02:08:43.090565 30296 tablet_service.cc:1514] Processing DeleteTablet for tablet e78b2e3b60c740b8993f0699bc3ade32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:43 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50172
I20240501 02:08:43.090644 30690 tablet_replica.cc:330] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: stopping tablet replica
I20240501 02:08:43.090615 30436 tablet_service.cc:1514] Processing DeleteTablet for tablet e78b2e3b60c740b8993f0699bc3ade32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:43 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57348
I20240501 02:08:43.090691 30690 raft_consensus.cc:2229] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:43.090732 30690 raft_consensus.cc:2258] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:43.090718 30692 tablet_replica.cc:330] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: stopping tablet replica
I20240501 02:08:43.090768 30692 raft_consensus.cc:2229] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:43.090828 30692 raft_consensus.cc:2258] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:43.090905 30691 tablet_replica.cc:330] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: stopping tablet replica
I20240501 02:08:43.090957 30691 raft_consensus.cc:2229] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:08:43.091033 30691 raft_consensus.cc:2258] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:43.091313 30690 ts_tablet_manager.cc:1875] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:43.091578 30692 ts_tablet_manager.cc:1875] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:43.092832 30690 ts_tablet_manager.cc:1888] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:43.092885 30690 log.cc:1196] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/e78b2e3b60c740b8993f0699bc3ade32
I20240501 02:08:43.093058 30690 ts_tablet_manager.cc:1909] T e78b2e3b60c740b8993f0699bc3ade32 P 69bde8c8748144f3941647218a198e61: Deleting consensus metadata
I20240501 02:08:43.093143 30692 ts_tablet_manager.cc:1888] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:43.093206 30692 log.cc:1196] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/e78b2e3b60c740b8993f0699bc3ade32
I20240501 02:08:43.093385 30144 catalog_manager.cc:4830] TS 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059): tablet e78b2e3b60c740b8993f0699bc3ade32 (table test-table [id=79d6dcdba45247d882525134dee78155]) successfully deleted
I20240501 02:08:43.093418 30692 ts_tablet_manager.cc:1909] T e78b2e3b60c740b8993f0699bc3ade32 P dbd56dd51583404b8a911564f7f35da9: Deleting consensus metadata
I20240501 02:08:43.093708 30144 catalog_manager.cc:4830] TS dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883): tablet e78b2e3b60c740b8993f0699bc3ade32 (table test-table [id=79d6dcdba45247d882525134dee78155]) successfully deleted
I20240501 02:08:43.093752 30691 ts_tablet_manager.cc:1875] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:43.095160 30691 ts_tablet_manager.cc:1888] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:08:43.095232 30691 log.cc:1196] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/e78b2e3b60c740b8993f0699bc3ade32
I20240501 02:08:43.095436 30691 ts_tablet_manager.cc:1909] T e78b2e3b60c740b8993f0699bc3ade32 P acfc992aecd1487f8d5dc32a90de815f: Deleting consensus metadata
I20240501 02:08:43.095794 30145 catalog_manager.cc:4830] TS acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703): tablet e78b2e3b60c740b8993f0699bc3ade32 (table test-table [id=79d6dcdba45247d882525134dee78155]) successfully deleted
I20240501 02:08:43.095933 30158 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:08:43.096076 30158 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:08:43.097393 30296 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4059f26b424c7aa508bd94892cfb67 (DEFAULT_TABLE table=test-table [id=d561f68379e04c9db16a805014122343]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:43.097393 30436 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4059f26b424c7aa508bd94892cfb67 (DEFAULT_TABLE table=test-table [id=d561f68379e04c9db16a805014122343]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:43.097496 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4059f26b424c7aa508bd94892cfb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:43.097497 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4059f26b424c7aa508bd94892cfb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:43.098340 30668 tablet_bootstrap.cc:492] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: Bootstrap starting.
I20240501 02:08:43.098676 30667 tablet_bootstrap.cc:492] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Bootstrap starting.
I20240501 02:08:43.098690 30668 tablet_bootstrap.cc:654] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:43.098753 30576 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4059f26b424c7aa508bd94892cfb67 (DEFAULT_TABLE table=test-table [id=d561f68379e04c9db16a805014122343]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:08:43.098843 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4059f26b424c7aa508bd94892cfb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:43.099027 30667 tablet_bootstrap.cc:654] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:43.099295 30668 tablet_bootstrap.cc:492] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: No bootstrap required, opened a new log
I20240501 02:08:43.099334 30668 ts_tablet_manager.cc:1367] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:43.099436 30668 raft_consensus.cc:348] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.099496 30668 raft_consensus.cc:374] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:43.099514 30668 raft_consensus.cc:729] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd56dd51583404b8a911564f7f35da9, State: Initialized, Role: FOLLOWER
I20240501 02:08:43.099552 30668 consensus_queue.cc:260] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.099634 30668 ts_tablet_manager.cc:1398] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:43.099730 30667 tablet_bootstrap.cc:492] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: No bootstrap required, opened a new log
I20240501 02:08:43.099771 30667 ts_tablet_manager.cc:1367] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:43.099901 30667 raft_consensus.cc:348] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.099962 30667 raft_consensus.cc:374] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:43.099979 30667 raft_consensus.cc:729] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Initialized, Role: FOLLOWER
I20240501 02:08:43.100030 30667 consensus_queue.cc:260] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.100129 30667 ts_tablet_manager.cc:1398] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:43.100172 30669 tablet_bootstrap.cc:492] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Bootstrap starting.
I20240501 02:08:43.100466 30669 tablet_bootstrap.cc:654] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:43.100973 30669 tablet_bootstrap.cc:492] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: No bootstrap required, opened a new log
I20240501 02:08:43.101013 30669 ts_tablet_manager.cc:1367] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:43.101138 30669 raft_consensus.cc:348] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.101197 30669 raft_consensus.cc:374] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:43.101213 30669 raft_consensus.cc:729] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bde8c8748144f3941647218a198e61, State: Initialized, Role: FOLLOWER
I20240501 02:08:43.101254 30669 consensus_queue.cc:260] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.101332 30669 ts_tablet_manager.cc:1398] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:43.111421 30743 raft_consensus.cc:482] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:08:43.111522 30743 raft_consensus.cc:504] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.111677 30743 leader_election.cc:288] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:43.111891 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4059f26b424c7aa508bd94892cfb67" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9" is_pre_election: true
I20240501 02:08:43.111927 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4059f26b424c7aa508bd94892cfb67" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61" is_pre_election: true
I20240501 02:08:43.111994 30596 raft_consensus.cc:2454] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 0.
I20240501 02:08:43.111991 30316 raft_consensus.cc:2454] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 0.
I20240501 02:08:43.112141 30388 leader_election.cc:302] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f; no voters:
I20240501 02:08:43.112248 30743 raft_consensus.cc:2781] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:08:43.112301 30743 raft_consensus.cc:482] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:08:43.112329 30743 raft_consensus.cc:3037] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:43.113196 30743 raft_consensus.cc:504] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.113341 30743 leader_election.cc:288] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 election: Requested vote from peers dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883), 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059)
I20240501 02:08:43.113497 30316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4059f26b424c7aa508bd94892cfb67" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd56dd51583404b8a911564f7f35da9"
I20240501 02:08:43.113505 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4059f26b424c7aa508bd94892cfb67" candidate_uuid: "acfc992aecd1487f8d5dc32a90de815f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69bde8c8748144f3941647218a198e61"
I20240501 02:08:43.113562 30316 raft_consensus.cc:3037] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:43.113576 30596 raft_consensus.cc:3037] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:43.114182 30596 raft_consensus.cc:2454] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 1.
I20240501 02:08:43.114281 30316 raft_consensus.cc:2454] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acfc992aecd1487f8d5dc32a90de815f in term 1.
I20240501 02:08:43.114374 30388 leader_election.cc:302] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69bde8c8748144f3941647218a198e61, acfc992aecd1487f8d5dc32a90de815f; no voters:
I20240501 02:08:43.114482 30743 raft_consensus.cc:2781] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:43.114574 30743 raft_consensus.cc:686] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 1 LEADER]: Becoming Leader. State: Replica: acfc992aecd1487f8d5dc32a90de815f, State: Running, Role: LEADER
I20240501 02:08:43.114624 30743 consensus_queue.cc:237] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } }
I20240501 02:08:43.115177 30158 catalog_manager.cc:5484] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f reported cstate change: term changed from 0 to 1, leader changed from <none> to acfc992aecd1487f8d5dc32a90de815f (127.22.12.66). New cstate: current_term: 1 leader_uuid: "acfc992aecd1487f8d5dc32a90de815f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acfc992aecd1487f8d5dc32a90de815f" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 46703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 } health_report { overall_health: UNKNOWN } } }
I20240501 02:08:43.126718 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.124419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:53645 (local address 127.22.12.65:45883)
0501 02:08:43.124470 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:43.124472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:43.124598 (+ 126us) server_negotiation.cc:407] Connection header received
0501 02:08:43.124663 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:43.124664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:43.124673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:43.124716 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:43.125062 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.125503 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.126325 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.126437 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.126454 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:43.126475 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:43.126514 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:43.126636 (+ 122us) server_negotiation.cc:299] Negotiation successful
0501 02:08:43.126659 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 02:08:43.126771 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.124361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.22.12.66:53645)
0501 02:08:43.124531 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:43.124541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:43.124602 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:43.124722 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:43.124723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:43.124724 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:08:43.125000 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.125002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.125568 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.125569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:43.126250 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.126252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.126327 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.126348 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:08:43.126578 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:08:43.126579 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:08:43.126625 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 02:08:43.126675 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20240501 02:08:43.128789 30596 raft_consensus.cc:1261] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Refusing update from remote peer acfc992aecd1487f8d5dc32a90de815f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:43.128811 30316 raft_consensus.cc:1261] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Refusing update from remote peer acfc992aecd1487f8d5dc32a90de815f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:08:43.129041 30743 consensus_queue.cc:1035] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [LEADER]: Connected to new peer: Peer: permanent_uuid: "69bde8c8748144f3941647218a198e61" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:43.129158 30673 consensus_queue.cc:1035] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd56dd51583404b8a911564f7f35da9" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:08:43.136344 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.133962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:46703 (local address 127.22.12.65:46603)
0501 02:08:43.134009 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:43.134021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:43.134062 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:43.134257 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:43.134259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:43.134260 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 02:08:43.134583 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.134586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.135457 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.135458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:43.135859 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:43.135860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.135917 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.135922 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:08:43.136218 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:08:43.136219 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:08:43.136258 (+ 39us) client_negotiation.cc:240] Negotiation successful
0501 02:08:43.136288 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20240501 02:08:43.136368 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.133992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:46603 (local address 127.22.12.66:46703)
0501 02:08:43.134036 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:43.134038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:43.134050 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:08:43.134130 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:43.134131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:43.134141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:43.134180 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:08:43.134666 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.135380 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:43.135933 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:43.136110 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:43.136129 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:43.136156 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:43.136222 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:43.136274 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 02:08:43.136296 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:08:43.144876 30158 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:43.144982 30158 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:08:43.145623 30158 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 364665fbd08849639523ca1ad85b5462: Sending DeleteTablet for 3 replicas of tablet 2c4059f26b424c7aa508bd94892cfb67
I20240501 02:08:43.145848 30436 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4059f26b424c7aa508bd94892cfb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:43 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:57348
I20240501 02:08:43.145859 30296 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4059f26b424c7aa508bd94892cfb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:43 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50172
I20240501 02:08:43.145972 30691 tablet_replica.cc:330] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: stopping tablet replica
I20240501 02:08:43.146031 30691 raft_consensus.cc:2229] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:08:43.146085 30691 raft_consensus.cc:2258] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:43.146165 30692 tablet_replica.cc:330] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: stopping tablet replica
I20240501 02:08:43.146225 30692 raft_consensus.cc:2229] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:43.146267 30692 raft_consensus.cc:2258] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:43.146276 30576 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4059f26b424c7aa508bd94892cfb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:08:43 UTC) from {username='oryx', principal='oryx/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:39250
I20240501 02:08:43.146360 30690 tablet_replica.cc:330] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: stopping tablet replica
I20240501 02:08:43.146410 30690 raft_consensus.cc:2229] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:08:43.146440 30690 raft_consensus.cc:2258] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:08:43.146616 30691 ts_tablet_manager.cc:1875] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:43.146660 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 30231
I20240501 02:08:43.146670 30692 ts_tablet_manager.cc:1875] T 2c4059f26b424c7aa508bd94892cfb67 P dbd56dd51583404b8a911564f7f35da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:43.148054 30691 ts_tablet_manager.cc:1888] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:08:43.148109 30691 log.cc:1196] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/2c4059f26b424c7aa508bd94892cfb67
I20240501 02:08:43.148296 30691 ts_tablet_manager.cc:1909] T 2c4059f26b424c7aa508bd94892cfb67 P acfc992aecd1487f8d5dc32a90de815f: Deleting consensus metadata
I20240501 02:08:43.148865 30690 ts_tablet_manager.cc:1875] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:08:43.149266 30145 catalog_manager.cc:4830] TS acfc992aecd1487f8d5dc32a90de815f (127.22.12.66:46703): tablet 2c4059f26b424c7aa508bd94892cfb67 (table test-table [id=d561f68379e04c9db16a805014122343]) successfully deleted
I20240501 02:08:43.149888 30690 ts_tablet_manager.cc:1888] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 02:08:43.149931 30690 log.cc:1196] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/2c4059f26b424c7aa508bd94892cfb67
I20240501 02:08:43.150060 30690 ts_tablet_manager.cc:1909] T 2c4059f26b424c7aa508bd94892cfb67 P 69bde8c8748144f3941647218a198e61: Deleting consensus metadata
I20240501 02:08:43.150475 30144 catalog_manager.cc:4830] TS 69bde8c8748144f3941647218a198e61 (127.22.12.67:41059): tablet 2c4059f26b424c7aa508bd94892cfb67 (table test-table [id=d561f68379e04c9db16a805014122343]) successfully deleted
W20240501 02:08:43.154956 30144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:45883 (error 108)
I20240501 02:08:43.155490 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:43.155311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45883 (local address 127.0.0.1:50204)
0501 02:08:43.155407 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:43.155443 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:45883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20240501 02:08:43.155527 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 30371
W20240501 02:08:43.156733 30144 catalog_manager.cc:4573] TS dbd56dd51583404b8a911564f7f35da9 (127.22.12.65:45883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c4059f26b424c7aa508bd94892cfb67: Network error: Client connection negotiation failed: client connection to 127.22.12.65:45883: connect: Connection refused (error 111)
I20240501 02:08:43.160495 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 30511
I20240501 02:08:43.165254 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 30127
2024-05-01T02:08:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2520 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:08:43 dist-test-slave-9tkh krb5kdc[30764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:08:43 dist-test-slave-9tkh krb5kdc[30764](info): set up 2 sockets
May 01 02:08:43 dist-test-slave-9tkh krb5kdc[30764](info): commencing operation
krb5kdc: starting...
W20240501 02:08:44.062397 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.877s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:08:44 dist-test-slave-9tkh krb5kdc[30764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:08:44.173945 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35341
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:44.252653 30777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:44.252863 30777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:44.252892 30777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:44.254276 30777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:44.254323 30777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:44.254335 30777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:44.254346 30777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:44.254356 30777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:44.256093 30777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:44.256302 30777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:44.256469 30777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:44.260681 30785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:44.261093 30782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:44.261426 30783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:44.261924 30777 server_base.cc:1034] running on GCE node
I20240501 02:08:44.262146 30777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:44.262377 30777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:44.263523 30777 hybrid_clock.cc:648] HybridClock initialized: now 1714529324263510 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2024-05-01T02:08:44Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1106 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:08:44 dist-test-slave-9tkh krb5kdc[30790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:08:44 dist-test-slave-9tkh krb5kdc[30790](info): set up 2 sockets
May 01 02:08:44 dist-test-slave-9tkh krb5kdc[30790](info): commencing operation
krb5kdc: starting...
W20240501 02:08:45.170408 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:08:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:08:45.283219 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:40073
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:40073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:45.363346 30803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:45.363575 30803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:45.363598 30803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:45.364933 30803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:45.364979 30803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:45.364989 30803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:45.365001 30803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:45.365010 30803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:45.366605 30803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:40073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:40073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:45.366808 30803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:45.366967 30803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:45.371105 30809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.371543 30811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.371984 30808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:45.372400 30803 server_base.cc:1034] running on GCE node
I20240501 02:08:45.372640 30803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:45.372969 30803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:45.374135 30803 hybrid_clock.cc:648] HybridClock initialized: now 1714529325374113 us; error 37 us; skew 500 ppm
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:45.376657 30803 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:08:45.377110 30803 webserver.cc:460] Webserver started at http://127.22.12.126:40249/ using document root <none> and password file <none>
I20240501 02:08:45.377287 30803 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:45.377328 30803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:45.377435 30803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:45.378407 30803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "895b5318574949a3a505de9eafaed73c"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "51c4b3f3a4e515b61aa8c9575973be48"
server_key_iv: "a3e17f00c7172f8490edd74d03959882"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.378772 30803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "895b5318574949a3a505de9eafaed73c"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "51c4b3f3a4e515b61aa8c9575973be48"
server_key_iv: "a3e17f00c7172f8490edd74d03959882"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.379951 30803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:45.380625 30818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.380795 30803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:45.380873 30803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "895b5318574949a3a505de9eafaed73c"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "51c4b3f3a4e515b61aa8c9575973be48"
server_key_iv: "a3e17f00c7172f8490edd74d03959882"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.380930 30803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:45.391047 30803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:45.393025 30803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:45.393143 30803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:45.396757 30803 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:40073
I20240501 02:08:45.396782 30870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:40073 every 8 connection(s)
I20240501 02:08:45.397086 30803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:45.397310 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 30803
I20240501 02:08:45.397404 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:08:45.397562 22577 external_mini_cluster.cc:1428] Setting key 7bee99d98ecf3f9c3082e37d73599462
I20240501 02:08:45.397869 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:08:45.400873 30871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Bootstrap starting.
I20240501 02:08:45.401428 30871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Neither blocks nor log segments found. Creating new log.
I20240501 02:08:45.401670 30871 log.cc:824] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:45.402427 30871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: No bootstrap required, opened a new log
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:45.404222 30871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:45.404367 30871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:08:45.404389 30871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895b5318574949a3a505de9eafaed73c, State: Initialized, Role: FOLLOWER
I20240501 02:08:45.404496 30871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:45.404575 30871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:45.404611 30871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:45.404660 30871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:08:45.405318 30871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:45.405428 30871 leader_election.cc:302] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 895b5318574949a3a505de9eafaed73c; no voters:
I20240501 02:08:45.405594 30871 leader_election.cc:288] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:08:45.405675 30876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:08:45.405831 30871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:45.405843 30876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 LEADER]: Becoming Leader. State: Replica: 895b5318574949a3a505de9eafaed73c, State: Running, Role: LEADER
I20240501 02:08:45.406037 30876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:45.406425 30878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "895b5318574949a3a505de9eafaed73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } } }
I20240501 02:08:45.406749 30878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:45.406684 30877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895b5318574949a3a505de9eafaed73c. Latest consensus state: current_term: 1 leader_uuid: "895b5318574949a3a505de9eafaed73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } } }
I20240501 02:08:45.406984 30877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:45.407132 30885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:45.407606 30885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:45.407907 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.398047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.22.12.126:40073)
0501 02:08:45.398214 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:45.398225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:45.398249 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:08:45.398892 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:45.398902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:45.398963 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:45.399129 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:45.399845 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.400953 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.401538 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.401790 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.403901 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:45.403910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:45.403910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:45.403936 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:45.405895 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.406299 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.406302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.406302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.406373 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.406605 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.406606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.406606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.406895 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:45.407099 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:45.407636 (+ 537us) server_negotiation.cc:299] Negotiation successful
0501 02:08:45.407766 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20240501 02:08:45.409015 30885 catalog_manager.cc:1347] Generated new cluster ID: a20cdda69b1b45ada289a2694c4e6b13
I20240501 02:08:45.409065 30885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:08:45.418296 30885 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:08:45.419126 30885 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:45.427517 30885 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Generated new TSK 0
I20240501 02:08:45.427788 30885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:08:45.459473 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40073
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:45.540102 30897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:45.540283 30897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:45.540305 30897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:45.541626 30897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:45.541676 30897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:45.541702 30897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:08:45.543402 30897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:45.543622 30897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:45.543817 30897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:45.548224 30903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.548650 30905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.549046 30902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:45.549707 30897 server_base.cc:1034] running on GCE node
I20240501 02:08:45.549892 30897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:45.550112 30897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:45.551249 30897 hybrid_clock.cc:648] HybridClock initialized: now 1714529325551231 us; error 31 us; skew 500 ppm
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:45.553805 30897 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:08:45.554205 30897 webserver.cc:460] Webserver started at http://127.22.12.65:33951/ using document root <none> and password file <none>
I20240501 02:08:45.554368 30897 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:45.554406 30897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:45.554505 30897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:45.555518 30897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "0dbce10a431e48768f8e798c8f2260dc"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "408fd5a0255d828c325759a6ead0c617"
server_key_iv: "be74bb887caee3b1c077ebc181ee582e"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.555816 30897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "0dbce10a431e48768f8e798c8f2260dc"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "408fd5a0255d828c325759a6ead0c617"
server_key_iv: "be74bb887caee3b1c077ebc181ee582e"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.557003 30897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:08:45.557618 30912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.557766 30897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:45.557840 30897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "0dbce10a431e48768f8e798c8f2260dc"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "408fd5a0255d828c325759a6ead0c617"
server_key_iv: "be74bb887caee3b1c077ebc181ee582e"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.557899 30897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:45.569571 30897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:45.571466 30897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:45.571581 30897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:45.571786 30897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:45.572085 30897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:45.572119 30897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.572149 30897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:45.572175 30897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.577376 30897 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38815
I20240501 02:08:45.577431 31025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38815 every 8 connection(s)
I20240501 02:08:45.577698 30897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:45.584008 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 30897
I20240501 02:08:45.584123 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:08:45.584306 22577 external_mini_cluster.cc:1428] Setting key 6aa5ff8a0f77a8a6187d738cc0faec3d
I20240501 02:08:45.586295 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.578346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:42179 (local address 127.22.12.126:40073)
0501 02:08:45.578396 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:45.578398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:45.578599 (+ 201us) server_negotiation.cc:407] Connection header received
0501 02:08:45.579055 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:45.579056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:45.579065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:45.579113 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:45.579819 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.580294 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.580782 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.580884 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.582942 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:45.582949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:45.582949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:45.582956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:45.584414 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.584943 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.584945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.584945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.584978 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.585284 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.585285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.585285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.585473 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:45.585589 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:45.586035 (+ 446us) server_negotiation.cc:299] Negotiation successful
0501 02:08:45.586181 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:08:45.586366 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.578320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:42179)
0501 02:08:45.578479 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.578504 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:45.578973 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:45.579162 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:45.579166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:45.579466 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:45.579750 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:45.579754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.580357 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.580358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:45.580728 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:45.580730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.581065 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.581517 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:45.581531 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:45.582850 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:45.584514 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.584521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:45.584524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:45.584786 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.585090 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.585091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:45.585091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:45.585199 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.585650 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:45.585656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:45.585863 (+ 207us) client_negotiation.cc:769] Sending connection context
0501 02:08:45.585989 (+ 126us) client_negotiation.cc:240] Negotiation successful
0501 02:08:45.586162 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20240501 02:08:45.586865 31026 heartbeater.cc:347] Connected to a master server at 127.22.12.126:40073
I20240501 02:08:45.586979 31026 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:45.587208 31026 heartbeater.cc:511] Master 127.22.12.126:40073 requested a full tablet report, sending...
I20240501 02:08:45.587949 30835 ts_manager.cc:194] Registered new tserver with Master: 0dbce10a431e48768f8e798c8f2260dc (127.22.12.65:38815)
I20240501 02:08:45.588721 30835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:42179
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:08:45.607146 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40073
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:45.684944 31031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:45.685144 31031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:45.685168 31031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:45.686491 31031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:45.686555 31031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:45.686581 31031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:08:45.688197 31031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:45.688393 31031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:45.688560 31031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:45.692862 31036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.693418 31037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.693662 31039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:45.694118 31031 server_base.cc:1034] running on GCE node
I20240501 02:08:45.694290 31031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:45.694492 31031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:45.695667 31031 hybrid_clock.cc:648] HybridClock initialized: now 1714529325695652 us; error 29 us; skew 500 ppm
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:45.698016 31031 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:08:45.698411 31031 webserver.cc:460] Webserver started at http://127.22.12.66:42809/ using document root <none> and password file <none>
I20240501 02:08:45.698626 31031 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:45.698671 31031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:45.698776 31031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:45.699738 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "31531642033c43e18b55431d2e65a516"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "8c3d3f7dffc7b381b4ee959f20aacbb4"
server_key_iv: "109281c234aa5a30a5feeeebf7735a9b"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.700016 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "31531642033c43e18b55431d2e65a516"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "8c3d3f7dffc7b381b4ee959f20aacbb4"
server_key_iv: "109281c234aa5a30a5feeeebf7735a9b"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.701156 31031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:08:45.701756 31046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.701900 31031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:08:45.701970 31031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "31531642033c43e18b55431d2e65a516"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "8c3d3f7dffc7b381b4ee959f20aacbb4"
server_key_iv: "109281c234aa5a30a5feeeebf7735a9b"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.702039 31031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:45.716323 31031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:45.718014 31031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:45.718127 31031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:45.718358 31031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:45.718673 31031 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:45.718708 31031 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.718730 31031 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:45.718753 31031 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.724073 31031 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:41385
I20240501 02:08:45.724135 31159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:41385 every 8 connection(s)
I20240501 02:08:45.724385 31031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:45.731549 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31031
I20240501 02:08:45.731669 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:08:45.731863 22577 external_mini_cluster.cc:1428] Setting key a6171557d5ed99ab9ec4bfb50a80e19e
I20240501 02:08:45.732672 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.725083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:48845 (local address 127.22.12.126:40073)
0501 02:08:45.725134 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:45.725137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:45.725307 (+ 170us) server_negotiation.cc:407] Connection header received
0501 02:08:45.725775 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:45.725776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:45.725788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:45.725841 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:45.726645 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.727126 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.727659 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.727828 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.729994 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:45.730000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:45.730000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:45.730007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:45.731277 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.731764 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.731765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.731766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.731797 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.732033 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.732034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.732034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.732158 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:45.732220 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:45.732495 (+ 275us) server_negotiation.cc:299] Negotiation successful
0501 02:08:45.732583 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 02:08:45.732823 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.725041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:48845)
0501 02:08:45.725201 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.725227 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:45.725675 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:45.725908 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:45.725911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:45.726216 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:45.726493 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:45.726496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.727207 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.727208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:45.727583 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:45.727584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.727981 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.728504 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:45.728523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:45.729893 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:45.731355 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.731361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:45.731364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:45.731642 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.731866 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.731867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:45.731867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:45.731957 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.732221 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:45.732228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:45.732360 (+ 132us) client_negotiation.cc:769] Sending connection context
0501 02:08:45.732463 (+ 103us) client_negotiation.cc:240] Negotiation successful
0501 02:08:45.732639 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20240501 02:08:45.733232 31160 heartbeater.cc:347] Connected to a master server at 127.22.12.126:40073
I20240501 02:08:45.733345 31160 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:45.733578 31160 heartbeater.cc:511] Master 127.22.12.126:40073 requested a full tablet report, sending...
I20240501 02:08:45.734012 30835 ts_manager.cc:194] Registered new tserver with Master: 31531642033c43e18b55431d2e65a516 (127.22.12.66:41385)
I20240501 02:08:45.734403 30835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:48845
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:08:45.754714 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40073
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:45.833585 31165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:45.833788 31165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:45.833811 31165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:45.835183 31165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:45.835233 31165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:45.835265 31165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:08:45.836925 31165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:45.837141 31165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:45.837342 31165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:45.841735 31171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.842092 31170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:45.842573 31173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:45.843055 31165 server_base.cc:1034] running on GCE node
I20240501 02:08:45.843273 31165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:45.843487 31165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:45.844633 31165 hybrid_clock.cc:648] HybridClock initialized: now 1714529325844618 us; error 31 us; skew 500 ppm
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:45.847046 31165 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:08:45.847457 31165 webserver.cc:460] Webserver started at http://127.22.12.67:34465/ using document root <none> and password file <none>
I20240501 02:08:45.847620 31165 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:45.847662 31165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:45.847766 31165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:08:45.848764 31165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "e1a2b2bed43c4b7288a34adf41534518"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "2ca3694e51d1a794a6b3559f8bb5ba62"
server_key_iv: "79c76c059452e42a804a7299756ea8e0"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.849056 31165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "e1a2b2bed43c4b7288a34adf41534518"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "2ca3694e51d1a794a6b3559f8bb5ba62"
server_key_iv: "79c76c059452e42a804a7299756ea8e0"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.850150 31165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 02:08:45.850783 31180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.850930 31165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:45.850999 31165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "e1a2b2bed43c4b7288a34adf41534518"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "2ca3694e51d1a794a6b3559f8bb5ba62"
server_key_iv: "79c76c059452e42a804a7299756ea8e0"
server_key_version: "encryptionkey@0"
I20240501 02:08:45.851061 31165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:45.860888 31165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:45.862869 31165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:45.862977 31165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:45.863170 31165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:45.863467 31165 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:45.863512 31165 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.863543 31165 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:45.863566 31165 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:45.868824 31165 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:35447
I20240501 02:08:45.868868 31293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:35447 every 8 connection(s)
I20240501 02:08:45.869140 31165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:08:45 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529325, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:08:45.876993 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.869791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:49657 (local address 127.22.12.126:40073)
0501 02:08:45.869849 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:45.869851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:45.870011 (+ 160us) server_negotiation.cc:407] Connection header received
0501 02:08:45.870480 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:45.870480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:45.870490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:45.870592 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:45.871318 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.871792 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.872398 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.872530 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.874747 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:45.874752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:45.874752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:45.874758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:45.876050 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.876374 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.876375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.876375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.876392 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.876524 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.876524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:45.876524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:45.876610 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:45.876653 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:45.876862 (+ 209us) server_negotiation.cc:299] Negotiation successful
0501 02:08:45.876921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:08:45.877108 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:45.869757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:49657)
0501 02:08:45.869904 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.869932 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:45.870398 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:45.870648 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:45.870652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:45.870951 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:45.871241 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:45.871245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.871895 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.871896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:45.872304 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:45.872306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.872658 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.873211 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:45.873229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:45.874661 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:08:45.876119 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.876122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:45.876125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:45.876318 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.876437 (+ 119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.876437 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:08:45.876438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:08:45.876486 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.876650 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:08:45.876652 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:08:45.876761 (+ 109us) client_negotiation.cc:769] Sending connection context
0501 02:08:45.876844 (+ 83us) client_negotiation.cc:240] Negotiation successful
0501 02:08:45.876947 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20240501 02:08:45.877599 31294 heartbeater.cc:347] Connected to a master server at 127.22.12.126:40073
I20240501 02:08:45.877704 31294 heartbeater.cc:464] Registering TS with master...
I20240501 02:08:45.877893 31294 heartbeater.cc:511] Master 127.22.12.126:40073 requested a full tablet report, sending...
I20240501 02:08:45.878261 30835 ts_manager.cc:194] Registered new tserver with Master: e1a2b2bed43c4b7288a34adf41534518 (127.22.12.67:35447)
I20240501 02:08:45.878690 30835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:49657
I20240501 02:08:45.879066 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31165
I20240501 02:08:45.879173 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:08:45.879354 22577 external_mini_cluster.cc:1428] Setting key 068943647bfb8dbe8c997fb5a19f9048
I20240501 02:08:45.880141 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab.
Entry for principal oryx/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab.
WARNING: no policy specified for oryx/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab.
Entry for principal oryx/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab.
WARNING: no policy specified for oryx/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab.
Entry for principal oryx/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab.
I20240501 02:08:45.944746 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 30897
I20240501 02:08:45.949090 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31031
I20240501 02:08:45.953164 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31165
I20240501 02:08:45.956694 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 30803
I20240501 02:08:45.961195 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:40073
--webserver_interface=127.22.12.126
--webserver_port=40249
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:40073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:08:46.039968 31303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:46.040179 31303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:46.040201 31303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:46.041549 31303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:08:46.041595 31303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:46.041608 31303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:46.041620 31303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:08:46.041630 31303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:08:46.043272 31303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:40073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:40073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=40249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:46.043488 31303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:46.043687 31303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:46.048105 31311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.048393 31308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.048887 31309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:46.049367 31303 server_base.cc:1034] running on GCE node
I20240501 02:08:46.049618 31303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:46.049865 31303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:46.051002 31303 hybrid_clock.cc:648] HybridClock initialized: now 1714529326050985 us; error 35 us; skew 500 ppm
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529326, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:46.053524 31303 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:08:46.053997 31303 webserver.cc:460] Webserver started at http://127.22.12.126:40249/ using document root <none> and password file <none>
I20240501 02:08:46.054163 31303 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:46.054219 31303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:46.055531 31303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:46.056138 31318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.056325 31303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:08:46.056404 31303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "895b5318574949a3a505de9eafaed73c"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "51c4b3f3a4e515b61aa8c9575973be48"
server_key_iv: "a3e17f00c7172f8490edd74d03959882"
server_key_version: "encryptionkey@0"
I20240501 02:08:46.056686 31303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:46.064725 31303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:46.066367 31303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:46.066479 31303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:46.070142 31303 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:40073
I20240501 02:08:46.070173 31370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:40073 every 8 connection(s)
I20240501 02:08:46.070495 31303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:08:46.071805 31371 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:08:46.072326 31371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Bootstrap starting.
I20240501 02:08:46.075304 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31303
I20240501 02:08:46.075608 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:38815
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:40073
--webserver_port=33951
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:08:46.081197 31371 log.cc:824] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:08:46.082846 31371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:08:46.083099 31371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Bootstrap complete.
I20240501 02:08:46.085103 31371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:46.085305 31371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895b5318574949a3a505de9eafaed73c, State: Initialized, Role: FOLLOWER
I20240501 02:08:46.085441 31371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:46.085517 31371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:08:46.085551 31371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:08:46.085598 31371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:08:46.086486 31371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:46.086606 31371 leader_election.cc:302] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 895b5318574949a3a505de9eafaed73c; no voters:
I20240501 02:08:46.086767 31371 leader_election.cc:288] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 02:08:46.086836 31375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 02:08:46.086994 31375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [term 2 LEADER]: Becoming Leader. State: Replica: 895b5318574949a3a505de9eafaed73c, State: Running, Role: LEADER
I20240501 02:08:46.086998 31371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:08:46.087105 31375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } }
I20240501 02:08:46.087402 31376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "895b5318574949a3a505de9eafaed73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } } }
I20240501 02:08:46.087471 31376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:46.087584 31376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895b5318574949a3a505de9eafaed73c. Latest consensus state: current_term: 2 leader_uuid: "895b5318574949a3a505de9eafaed73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895b5318574949a3a505de9eafaed73c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40073 } } }
I20240501 02:08:46.087627 31376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c [sys.catalog]: This master's current role is: LEADER
I20240501 02:08:46.087895 31380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:08:46.088404 31380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:08:46.088508 31380 catalog_manager.cc:1259] Loaded cluster ID: a20cdda69b1b45ada289a2694c4e6b13
I20240501 02:08:46.088737 31380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:08:46.090016 31380 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:08:46.090193 31380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 895b5318574949a3a505de9eafaed73c: Loaded TSK: 0
I20240501 02:08:46.090384 31380 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 02:08:46.156543 31373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:46.156733 31373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:46.156754 31373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:46.158061 31373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:46.158107 31373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:46.158131 31373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:08:46.159854 31373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.65.keytab
--principal=oryx/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:38815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=33951
--tserver_master_addrs=127.22.12.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:46.160068 31373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:46.160257 31373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:46.165395 31400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.165751 31398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.166178 31397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:46.166749 31373 server_base.cc:1034] running on GCE node
I20240501 02:08:46.166975 31373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:46.167212 31373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:46.168385 31373 hybrid_clock.cc:648] HybridClock initialized: now 1714529326168365 us; error 38 us; skew 500 ppm
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529326, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:46.171169 31373 init.cc:377] Logged in from keytab as oryx/127.22.12.65@KRBTEST.COM (short username oryx)
I20240501 02:08:46.171702 31373 webserver.cc:460] Webserver started at http://127.22.12.65:33951/ using document root <none> and password file <none>
I20240501 02:08:46.171914 31373 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:46.171988 31373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:46.173669 31373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:46.174311 31407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.174496 31373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.174623 31373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "0dbce10a431e48768f8e798c8f2260dc"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "408fd5a0255d828c325759a6ead0c617"
server_key_iv: "be74bb887caee3b1c077ebc181ee582e"
server_key_version: "encryptionkey@0"
I20240501 02:08:46.175021 31373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:46.190688 31373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:46.192859 31373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:46.192975 31373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:46.193169 31373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:46.193449 31373 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:46.193480 31373 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.193511 31373 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:46.193535 31373 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.198948 31373 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38815
I20240501 02:08:46.198988 31520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38815 every 8 connection(s)
I20240501 02:08:46.199344 31373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:08:46.200093 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31373
I20240501 02:08:46.200397 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:41385
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:40073
--webserver_port=42809
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.207567 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.200163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:55295)
0501 02:08:46.200310 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.200341 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.200904 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.201276 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.201280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.201621 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.202012 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.202019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.203585 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.203586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.204004 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.204006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.204355 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.204940 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.204953 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.207361 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20240501 02:08:46.207671 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.200422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:55295 (local address 127.22.12.126:40073)
0501 02:08:46.200842 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.200853 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.200882 (+ 29us) server_negotiation.cc:407] Connection header received
0501 02:08:46.201088 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.201092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.201130 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.201241 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.202155 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.203445 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.204091 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.204356 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.207527 (+ 3171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:55295: BlockingRecv error: recv got EOF from 127.22.12.65:55295 (error 108)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":97,"threads_started":1}
W20240501 02:08:46.208216 31521 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:40073 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.12.126:40073: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.213415 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.208471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:40591)
0501 02:08:46.208593 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.208603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.208657 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.208801 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.208802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.208950 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.209166 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.209169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.210060 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.210061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.210663 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.210665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.210741 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.211289 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.211295 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.213336 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20240501 02:08:46.213759 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.208540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:40591 (local address 127.22.12.126:40073)
0501 02:08:46.208590 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.208592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.208640 (+ 48us) server_negotiation.cc:407] Connection header received
0501 02:08:46.208708 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.208709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.208720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.208783 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.209257 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.209974 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.210743 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.210912 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.213704 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:40591: BlockingRecv error: recv got EOF from 127.22.12.65:40591 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.218858 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.214270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:39863)
0501 02:08:46.214317 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.214327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.214361 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.214492 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.214493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.214651 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.214817 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.214820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.215683 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.215684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.216203 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.216204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.216277 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.216887 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.216892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.218789 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 02:08:46.218961 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.214317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:39863 (local address 127.22.12.126:40073)
0501 02:08:46.214351 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.214353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.214363 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:08:46.214410 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.214411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.214420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.214480 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.214897 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.215604 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.216283 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.216473 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.218897 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:39863: BlockingRecv error: recv got EOF from 127.22.12.65:39863 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20240501 02:08:46.219156 31521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 02:08:46.284523 31525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:46.284732 31525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:46.284756 31525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:46.286159 31525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:46.286212 31525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:46.286239 31525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:08:46.288002 31525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.66.keytab
--principal=oryx/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:41385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=42809
--tserver_master_addrs=127.22.12.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:46.288223 31525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:46.288400 31525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:46.292757 31531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.293120 31533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.293592 31530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:46.294067 31525 server_base.cc:1034] running on GCE node
I20240501 02:08:46.294250 31525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:46.294461 31525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:46.295596 31525 hybrid_clock.cc:648] HybridClock initialized: now 1714529326295593 us; error 40 us; skew 500 ppm
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529326, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:46.297912 31525 init.cc:377] Logged in from keytab as oryx/127.22.12.66@KRBTEST.COM (short username oryx)
I20240501 02:08:46.298275 31525 webserver.cc:460] Webserver started at http://127.22.12.66:42809/ using document root <none> and password file <none>
I20240501 02:08:46.298422 31525 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:46.298467 31525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:46.299700 31525 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:08:46.300171 31540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.300340 31525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.300418 31525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "31531642033c43e18b55431d2e65a516"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "8c3d3f7dffc7b381b4ee959f20aacbb4"
server_key_iv: "109281c234aa5a30a5feeeebf7735a9b"
server_key_version: "encryptionkey@0"
I20240501 02:08:46.300693 31525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:46.313321 31525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:46.315011 31525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:46.315110 31525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:46.315295 31525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:46.315599 31525 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:46.315635 31525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.315657 31525 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:46.315680 31525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.321019 31525 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:41385
I20240501 02:08:46.321086 31653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:41385 every 8 connection(s)
I20240501 02:08:46.321368 31525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:08:46.325791 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31525
I20240501 02:08:46.326032 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:35447
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:40073
--webserver_port=34465
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.328083 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.322020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:55719)
0501 02:08:46.322161 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.322186 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.322671 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.322905 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.322909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.323270 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.323697 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.323703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.324507 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.324508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.324890 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.324891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.325228 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.325744 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.325760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.327909 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
W20240501 02:08:46.328074 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.322034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:55719 (local address 127.22.12.126:40073)
0501 02:08:46.322080 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.322081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.322270 (+ 189us) server_negotiation.cc:407] Connection header received
0501 02:08:46.322771 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.322772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.322789 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.322836 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.323791 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.324426 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.324953 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.325055 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.328022 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:55719: BlockingRecv error: recv got EOF from 127.22.12.66:55719 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240501 02:08:46.328300 31654 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:40073 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.12.126:40073: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.332253 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.328498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:36763)
0501 02:08:46.328543 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.328553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.328588 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.328718 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.328719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.328845 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.329039 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.329041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.329642 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.329643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.330014 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.330015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.330074 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.330619 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.330622 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.332204 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20240501 02:08:46.332391 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.328561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:36763 (local address 127.22.12.126:40073)
0501 02:08:46.328621 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.328623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.328631 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:46.328654 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.328655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.328665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.328707 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.329111 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.329574 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.330087 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.330246 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.332329 (+ 2083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:36763: BlockingRecv error: recv got EOF from 127.22.12.66:36763 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.336196 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.332594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:36525)
0501 02:08:46.332636 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.332646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.332680 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.332807 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.332808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.332919 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.333033 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.333034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.333619 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.333620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.333958 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.333959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.334018 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.334503 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.334506 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.336130 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20240501 02:08:46.336287 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.332652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:36525 (local address 127.22.12.126:40073)
0501 02:08:46.332711 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.332713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.332721 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:46.332740 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.332741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.332751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.332798 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.333098 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.333554 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.334028 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.334176 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.336223 (+ 2047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:36525: BlockingRecv error: recv got EOF from 127.22.12.66:36525 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20240501 02:08:46.336390 31654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 02:08:46.405494 31657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:08:46.405694 31657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:08:46.405716 31657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:08:46.407069 31657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:08:46.407119 31657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:08:46.407143 31657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:08:46.408779 31657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/krb5kdc/oryx_127.22.12.67.keytab
--principal=oryx/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:35447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=34465
--tserver_master_addrs=127.22.12.126:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:08:46.408999 31657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:08:46.409185 31657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:08:46.413483 31662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.413944 31663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:08:46.414425 31665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:08:46.414836 31657 server_base.cc:1034] running on GCE node
I20240501 02:08:46.415031 31657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:08:46.415261 31657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:08:46.416390 31657 hybrid_clock.cc:648] HybridClock initialized: now 1714529326416372 us; error 33 us; skew 500 ppm
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529326, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:08:46.418733 31657 init.cc:377] Logged in from keytab as oryx/127.22.12.67@KRBTEST.COM (short username oryx)
I20240501 02:08:46.419131 31657 webserver.cc:460] Webserver started at http://127.22.12.67:34465/ using document root <none> and password file <none>
I20240501 02:08:46.419312 31657 fs_manager.cc:359] Metadata directory not provided
I20240501 02:08:46.419379 31657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:08:46.420825 31657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:08:46.421326 31672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.421478 31657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.421546 31657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "e1a2b2bed43c4b7288a34adf41534518"
format_stamp: "Formatted at 2024-05-01 02:08:45 on dist-test-slave-9tkh"
server_key: "2ca3694e51d1a794a6b3559f8bb5ba62"
server_key_iv: "79c76c059452e42a804a7299756ea8e0"
server_key_version: "encryptionkey@0"
I20240501 02:08:46.421825 31657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:08:46.429201 31657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:08:46.430984 31657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:08:46.431087 31657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:08:46.431309 31657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:08:46.431597 31657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:08:46.431627 31657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.431650 31657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:08:46.431674 31657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:08:46.437000 31657 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:35447
I20240501 02:08:46.437052 31785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:35447 every 8 connection(s)
I20240501 02:08:46.437368 31657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:08:46.440479 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31657
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 02:08:46.443895 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.438072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:39983 (local address 127.22.12.126:40073)
0501 02:08:46.438150 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.438152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.438278 (+ 126us) server_negotiation.cc:407] Connection header received
0501 02:08:46.438790 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.438791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.438802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.438851 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.439557 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.440020 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.440516 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.440680 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.443837 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:39983: BlockingRecv error: recv got EOF from 127.22.12.67:39983 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 02:08:46.443928 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.438035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:39983)
0501 02:08:46.438174 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.438200 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.438702 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.438898 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.438901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.439207 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.439483 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.439488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.440076 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.440076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.440440 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.440441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.440842 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.441428 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.441450 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.443769 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20240501 02:08:46.444167 31786 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:40073 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.12.126:40073: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.447548 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.440992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.22.12.126:40073)
0501 02:08:46.441125 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.441127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.441135 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:46.441156 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.441156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.441170 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.441234 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.442099 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.442794 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.443381 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.443563 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.444648 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:08:46.444654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:08:46.444655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:08:46.444675 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:08:46.446316 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:46.446665 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:46.446668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:46.446668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:46.446713 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:46.446862 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:08:46.446862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:08:46.446863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:08:46.447092 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:46.447232 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0501 02:08:46.447336 (+ 104us) server_negotiation.cc:299] Negotiation successful
0501 02:08:46.447462 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.448184 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.444348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:38569)
0501 02:08:46.444400 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.444412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.444457 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.444561 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.444562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.444678 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.444830 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.444832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.445475 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.445477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.445832 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.445833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.445891 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.446299 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.446302 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.448123 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 02:08:46.448266 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.444411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:38569 (local address 127.22.12.126:40073)
0501 02:08:46.444459 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.444460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.444468 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:46.444506 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.444507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.444514 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.444554 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.444905 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.445406 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.445901 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.446047 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.448202 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:38569: BlockingRecv error: recv got EOF from 127.22.12.67:38569 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:46 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:46.451615 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:46.448527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:45807)
0501 02:08:46.448568 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.448577 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:46.448612 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.448713 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:46.448714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:46.448827 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:46.448939 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.448940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.449492 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.449493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:46.449848 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:46.449849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.449907 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.450260 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:46.450263 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:46.451568 (+ 1305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20240501 02:08:46.451689 31524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:46.448583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:45807 (local address 127.22.12.126:40073)
0501 02:08:46.448621 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:46.448623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:46.448631 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:46.448650 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:46.448651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:46.448659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.448709 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:46.449002 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.449431 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.449921 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.450046 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.451648 (+ 1602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:45807: BlockingRecv error: recv got EOF from 127.22.12.67:45807 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240501 02:08:46.451735 31786 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 01 02:08:47 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:47 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:47.223969 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:47.219613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:33477)
0501 02:08:47.219739 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.219754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:47.219837 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:47.220009 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:47.220010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:47.220167 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:47.220352 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:47.220354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.221010 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:47.221012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:47.221413 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:47.221414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.221478 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:47.221949 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:47.221952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:47.223863 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
W20240501 02:08:47.224066 31792 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:47.219713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:33477 (local address 127.22.12.126:40073)
0501 02:08:47.219870 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:47.219872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:47.219881 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:47.219907 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:47.219908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:47.219917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:47.219976 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:47.220421 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.220913 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:47.221476 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.221584 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:47.224001 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:33477: BlockingRecv error: recv got EOF from 127.22.12.65:33477 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
May 01 02:08:47 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:47 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:47.340950 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:47.336851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:41571)
0501 02:08:47.337019 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.337030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:47.337106 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:47.337273 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:47.337274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:47.337435 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:47.337620 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:47.337622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.338269 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:47.338270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:47.338714 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:47.338717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.338777 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:47.339262 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:47.339266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:47.340881 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20240501 02:08:47.341079 31792 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:47.336930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:41571 (local address 127.22.12.126:40073)
0501 02:08:47.336985 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:47.336987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:47.337149 (+ 162us) server_negotiation.cc:407] Connection header received
0501 02:08:47.337182 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:47.337183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:47.337195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:47.337241 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:47.337706 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.338195 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:47.338811 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.338952 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:47.341015 (+ 2063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:41571: BlockingRecv error: recv got EOF from 127.22.12.66:41571 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
May 01 02:08:47 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:47 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:47.456116 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:47.452126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:48425)
0501 02:08:47.452256 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.452268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:47.452327 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:47.452452 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:47.452453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:47.452619 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:47.452810 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:47.452812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.453441 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:47.453442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:47.453818 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:47.453819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.453877 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:47.454364 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:47.454368 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:47.456050 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20240501 02:08:47.456229 31792 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:47.452166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:48425 (local address 127.22.12.126:40073)
0501 02:08:47.452225 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:47.452227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:47.452317 (+ 90us) server_negotiation.cc:407] Connection header received
0501 02:08:47.452383 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:47.452383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:47.452396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:47.452437 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:47.452886 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.453367 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:47.453886 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:47.454046 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:47.456148 (+ 2102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:48425: BlockingRecv error: recv got EOF from 127.22.12.67:48425 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
May 01 02:08:48 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:48 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:48.228350 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:48.224461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:51031)
0501 02:08:48.224606 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:48.224616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:48.224684 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:48.224816 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:48.224817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:48.224973 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:48.225128 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:48.225130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.225861 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:48.225862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:48.226234 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:48.226236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.226292 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:48.226776 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:48.226779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:48.228290 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
W20240501 02:08:48.228466 31796 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:48.224544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:51031 (local address 127.22.12.126:40073)
0501 02:08:48.224634 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:48.224636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:48.224667 (+ 31us) server_negotiation.cc:407] Connection header received
0501 02:08:48.224741 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:48.224742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:48.224752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:48.224807 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:48.225222 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.225784 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:48.226296 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.226406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:48.228400 (+ 1994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:51031: BlockingRecv error: recv got EOF from 127.22.12.65:51031 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":47,"threads_started":1}
May 01 02:08:48 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:48 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:48.345448 31797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:48.341432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:60977)
0501 02:08:48.341546 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:48.341556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:48.341615 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:48.341763 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:48.341763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:48.341933 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:48.342088 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:48.342090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.342777 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:48.342778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:48.343147 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:48.343148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.343207 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:48.343745 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:48.343750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:48.345387 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20240501 02:08:48.345597 31796 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:48.341498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:60977 (local address 127.22.12.126:40073)
0501 02:08:48.341542 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:48.341544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:48.341622 (+ 78us) server_negotiation.cc:407] Connection header received
0501 02:08:48.341678 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:48.341678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:48.341689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:48.341743 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:48.342153 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.342706 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:48.343240 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.343401 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:48.345510 (+ 2109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:60977: BlockingRecv error: recv got EOF from 127.22.12.66:60977 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
May 01 02:08:48 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:48 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:48.460597 31798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:48.456626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:45089)
0501 02:08:48.456789 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:48.456801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:48.456862 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:48.456991 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:48.456991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:48.457148 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:48.457303 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:48.457305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.457876 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:48.457877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:48.458232 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:48.458233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.458283 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:48.458806 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:48.458810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:48.460537 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
W20240501 02:08:48.460685 31796 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:48.456779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:45089 (local address 127.22.12.126:40073)
0501 02:08:48.456833 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:48.456835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:48.456846 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:08:48.456915 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:48.456916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:48.456929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:48.456977 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:48.457358 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.457820 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:48.458288 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:48.458398 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:48.460626 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:45089: BlockingRecv error: recv got EOF from 127.22.12.67:45089 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
May 01 02:08:49 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:49 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:49.232698 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:49.228827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:55315)
0501 02:08:49.228946 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:49.228955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:49.229004 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:49.229131 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:49.229132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:49.229285 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:49.229437 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:49.229438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.230076 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:49.230077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:49.230437 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:49.230438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.230488 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:49.230966 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:49.230969 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:49.232611 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
W20240501 02:08:49.232805 31800 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:49.228879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:55315 (local address 127.22.12.126:40073)
0501 02:08:49.229004 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:49.229006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:49.229040 (+ 34us) server_negotiation.cc:407] Connection header received
0501 02:08:49.229061 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:49.229062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:49.229072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:49.229121 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:49.229507 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.230017 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:49.230490 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.230650 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:49.232743 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:55315: BlockingRecv error: recv got EOF from 127.22.12.65:55315 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
May 01 02:08:49 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:49 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:49.349759 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:49.345959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:41563)
0501 02:08:49.346093 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:49.346102 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:49.346160 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:49.346280 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:49.346281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:49.346435 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:49.346624 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:49.346626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.347230 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:49.347231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:49.347599 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:49.347600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.347653 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:49.348112 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:49.348116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:49.349702 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
W20240501 02:08:49.349884 31800 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:49.346025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:41563 (local address 127.22.12.126:40073)
0501 02:08:49.346092 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:49.346094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:49.346156 (+ 62us) server_negotiation.cc:407] Connection header received
0501 02:08:49.346217 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:49.346218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:49.346228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:49.346279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:49.346680 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.347166 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:49.347657 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.347763 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:49.349819 (+ 2056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:41563: BlockingRecv error: recv got EOF from 127.22.12.66:41563 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
May 01 02:08:49 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:49 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:49.465242 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:49.461032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:48651)
0501 02:08:49.461184 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:49.461196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:49.461277 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:49.461416 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:49.461416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:49.461609 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:49.461761 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:49.461763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.462400 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:49.462401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:49.462812 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:49.462814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.462880 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:49.463448 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:49.463453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:49.465183 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
W20240501 02:08:49.465344 31800 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:49.461115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:48651 (local address 127.22.12.126:40073)
0501 02:08:49.461188 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:49.461190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:49.461262 (+ 72us) server_negotiation.cc:407] Connection header received
0501 02:08:49.461347 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:49.461347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:49.461360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:49.461398 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:49.461841 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.462327 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:49.462881 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:49.463071 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:49.465274 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:48651: BlockingRecv error: recv got EOF from 127.22.12.67:48651 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
May 01 02:08:50 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:50 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:50.237491 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:50.233177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:50543)
0501 02:08:50.233315 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.233329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:50.233408 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:50.233543 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:50.233544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:50.233719 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:50.233878 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:50.233879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.234669 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:50.234671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:50.235059 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:50.235061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.235128 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:50.235619 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:50.235622 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:50.237410 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
W20240501 02:08:50.237615 31804 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:50.233268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:50543 (local address 127.22.12.126:40073)
0501 02:08:50.233434 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:50.233436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:50.233444 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:50.233466 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:50.233467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:50.233477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:50.233532 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:50.233974 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.234583 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:50.235130 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.235243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:50.237552 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:50543: BlockingRecv error: recv got EOF from 127.22.12.65:50543 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
May 01 02:08:50 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:50 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:50.354249 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:50.350262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:53011)
0501 02:08:50.350404 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.350417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:50.350502 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:50.350675 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:50.350676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:50.350866 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:50.351020 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:50.351022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.351657 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:50.351658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:50.352017 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:50.352018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.352079 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:50.352565 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:50.352569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:50.354187 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20240501 02:08:50.354357 31804 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:50.350354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:53011 (local address 127.22.12.126:40073)
0501 02:08:50.350432 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:50.350435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:50.350493 (+ 58us) server_negotiation.cc:407] Connection header received
0501 02:08:50.350599 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:50.350600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:50.350613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:50.350662 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:50.351102 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.351581 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:50.352092 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.352252 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:50.354283 (+ 2031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:53011: BlockingRecv error: recv got EOF from 127.22.12.66:53011 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
May 01 02:08:50 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:50 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:50.469872 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:50.465712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:57361)
0501 02:08:50.465852 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.465862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:50.465933 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:50.466057 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:50.466057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:50.466225 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:50.466385 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:50.466386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.467063 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:50.467064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:50.467434 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:50.467435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.467497 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:50.467999 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:50.468003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:50.469810 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20240501 02:08:50.469974 31804 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:50.465780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:57361 (local address 127.22.12.126:40073)
0501 02:08:50.465834 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:50.465837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:50.465926 (+ 89us) server_negotiation.cc:407] Connection header received
0501 02:08:50.465986 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:50.465987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:50.466000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:50.466045 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:50.466466 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.466987 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:50.467509 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:50.467667 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:50.469904 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:57361: BlockingRecv error: recv got EOF from 127.22.12.67:57361 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 01 02:08:51 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:51 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:51.245985 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:51.238020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:35353)
0501 02:08:51.238186 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:51.238198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:51.238262 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:51.238392 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:51.238393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:51.238600 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:51.238762 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:51.238765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.239397 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:51.239398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:51.239763 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:51.239764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.239824 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:51.244144 (+ 4320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:51.244149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:51.245905 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
W20240501 02:08:51.246101 31808 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:51.238088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:35353 (local address 127.22.12.126:40073)
0501 02:08:51.238223 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:51.238225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:51.238247 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:08:51.238313 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:51.238313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:51.238323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:51.238380 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:51.238839 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.239337 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:51.239828 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.239932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:51.246042 (+ 6110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:35353: BlockingRecv error: recv got EOF from 127.22.12.65:35353 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
May 01 02:08:51 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:51 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:51.358644 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:51.354722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:32975)
0501 02:08:51.354863 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:51.354875 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:51.354938 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:51.355070 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:51.355071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:51.355245 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:51.355404 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:51.355405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.356042 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:51.356044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:51.356417 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:51.356418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.356476 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:51.356957 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:51.356960 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:51.358568 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
W20240501 02:08:51.358703 31808 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:51.354826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:32975 (local address 127.22.12.126:40073)
0501 02:08:51.354867 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:51.354870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:51.354918 (+ 48us) server_negotiation.cc:407] Connection header received
0501 02:08:51.355000 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:51.355000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:51.355011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:51.355055 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:51.355482 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.355971 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:51.356491 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.356649 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:51.358652 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:32975: BlockingRecv error: recv got EOF from 127.22.12.66:32975 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
May 01 02:08:51 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:51 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:51.474329 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:51.470344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:41157)
0501 02:08:51.470470 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:51.470480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:51.470579 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:51.470704 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:51.470705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:51.470861 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:51.471018 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:51.471020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.471601 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:51.471602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:51.471959 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:51.471960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.472010 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:51.472489 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:51.472493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:51.474257 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
W20240501 02:08:51.474423 31808 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:51.470395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:41157 (local address 127.22.12.126:40073)
0501 02:08:51.470450 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:51.470453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:51.470555 (+ 102us) server_negotiation.cc:407] Connection header received
0501 02:08:51.470630 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:51.470631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:51.470643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:51.470689 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:51.471076 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.471543 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:51.472013 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:51.472114 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:51.474364 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:41157: BlockingRecv error: recv got EOF from 127.22.12.67:41157 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 01 02:08:52 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:52 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:52.250589 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:52.246569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:51719)
0501 02:08:52.246723 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:52.246734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:52.246795 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:52.246916 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:52.246916 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:52.247079 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:52.247233 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:52.247235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.247904 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:52.247906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:52.248332 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:52.248333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.248398 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:52.248880 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:52.248883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:52.250469 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
W20240501 02:08:52.250670 31812 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:52.246675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:51719 (local address 127.22.12.126:40073)
0501 02:08:52.246783 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:52.246785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:52.246807 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:08:52.246833 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:52.246834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:52.246844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:52.246903 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:52.247301 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.247810 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:52.248400 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.248509 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:52.250608 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:51719: BlockingRecv error: recv got EOF from 127.22.12.65:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":47,"threads_started":1}
May 01 02:08:52 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:52 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:52.363423 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:52.359258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:50195)
0501 02:08:52.359406 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:52.359417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:52.359503 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:52.359705 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:52.359706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:52.359873 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:52.360027 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:52.360029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.360716 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:52.360717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:52.361117 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:52.361118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.361181 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:52.361690 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:52.361694 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:52.363321 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
W20240501 02:08:52.363483 31812 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:52.359360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:50195 (local address 127.22.12.126:40073)
0501 02:08:52.359414 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:52.359416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:52.359482 (+ 66us) server_negotiation.cc:407] Connection header received
0501 02:08:52.359584 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:52.359585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:52.359597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:52.359638 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:52.360095 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.360634 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:52.361181 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.361297 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:52.363432 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:50195: BlockingRecv error: recv got EOF from 127.22.12.66:50195 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
May 01 02:08:52 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:52 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:52.478905 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:52.474863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:59659)
0501 02:08:52.475021 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:52.475031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:52.475092 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:52.475232 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:52.475233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:52.475418 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:52.475576 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:52.475577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.476186 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:52.476186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:52.476581 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:52.476583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.476639 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:52.477116 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:52.477119 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:52.478823 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
W20240501 02:08:52.478978 31812 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:52.474948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:59659 (local address 127.22.12.126:40073)
0501 02:08:52.475008 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:52.475010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:52.475089 (+ 79us) server_negotiation.cc:407] Connection header received
0501 02:08:52.475160 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:52.475160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:52.475173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:52.475213 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:52.475645 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.476117 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:52.476658 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:52.476781 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:52.478916 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:59659: BlockingRecv error: recv got EOF from 127.22.12.67:59659 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 01 02:08:53 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:53 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:53.255190 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:53.251211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:49907)
0501 02:08:53.251339 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:53.251348 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:53.251399 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:53.251542 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:53.251543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:53.251705 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:53.251859 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:53.251861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.252486 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:53.252486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:53.252872 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:53.252874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.252930 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:53.253426 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:53.253430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:53.255111 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20240501 02:08:53.255290 31816 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:53.251289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:49907 (local address 127.22.12.126:40073)
0501 02:08:53.251423 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:53.251424 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:53.251433 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:53.251458 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:53.251459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:53.251469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:53.251534 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:53.251920 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.252423 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:53.252939 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.253055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:53.255233 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:49907: BlockingRecv error: recv got EOF from 127.22.12.65:49907 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
May 01 02:08:53 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:53 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:53.368096 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:53.363976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:33875)
0501 02:08:53.364135 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:53.364147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:53.364223 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:53.364372 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:53.364373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:53.364541 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:53.364695 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:53.364697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.365392 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:53.365393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:53.365907 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:53.365909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.365969 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:53.366429 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:53.366432 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:53.368020 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
W20240501 02:08:53.368175 31816 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:53.364072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:33875 (local address 127.22.12.126:40073)
0501 02:08:53.364139 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:53.364142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:53.364214 (+ 72us) server_negotiation.cc:407] Connection header received
0501 02:08:53.364297 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:53.364298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:53.364309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:53.364357 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:53.364776 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.365310 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:53.365975 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.366097 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:53.368122 (+ 2025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:33875: BlockingRecv error: recv got EOF from 127.22.12.66:33875 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
May 01 02:08:53 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:53 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:53.483409 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:53.479485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:33407)
0501 02:08:53.479646 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:53.479655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:53.479717 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:53.479849 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:53.479849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:53.480015 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:53.480172 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:53.480173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.480866 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:53.480867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:53.481242 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:53.481243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.481302 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:53.481772 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:53.481776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:53.483349 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
W20240501 02:08:53.483505 31816 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:53.479553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:33407 (local address 127.22.12.126:40073)
0501 02:08:53.479630 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:53.479632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:53.479711 (+ 79us) server_negotiation.cc:407] Connection header received
0501 02:08:53.479761 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:53.479762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:53.479776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:53.479836 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:53.480245 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.480797 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:53.481318 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:53.481438 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:53.483446 (+ 2008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:33407: BlockingRecv error: recv got EOF from 127.22.12.67:33407 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 02:08:54 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:54 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:54.259781 31819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:54.255728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:33753)
0501 02:08:54.255874 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:54.255884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:54.255931 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:54.256075 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:54.256076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:54.256228 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:54.256383 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:54.256385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.257036 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:54.257037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:54.257420 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:54.257422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.257505 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:54.257997 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:54.258001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:54.259721 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
W20240501 02:08:54.259918 31820 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:54.255841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:33753 (local address 127.22.12.126:40073)
0501 02:08:54.255956 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:54.255958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:54.255966 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:08:54.255988 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:54.255988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:54.255998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:54.256061 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:54.256444 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.256945 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:54.257500 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.257603 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:54.259842 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:33753: BlockingRecv error: recv got EOF from 127.22.12.65:33753 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
May 01 02:08:54 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:54 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:54.372785 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:54.368628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:47005)
0501 02:08:54.368753 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:54.368763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:54.368830 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:54.368978 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:54.368978 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:54.369131 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:54.369287 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:54.369289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.370146 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:54.370147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:54.370500 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:54.370501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.370601 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:54.371063 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:54.371067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:54.372701 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20240501 02:08:54.372870 31820 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:54.368722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:47005 (local address 127.22.12.126:40073)
0501 02:08:54.368804 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:54.368806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:54.368821 (+ 15us) server_negotiation.cc:407] Connection header received
0501 02:08:54.368893 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:54.368894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:54.368904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:54.368967 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:54.369359 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.370058 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:54.370615 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.370738 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:54.372810 (+ 2072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:47005: BlockingRecv error: recv got EOF from 127.22.12.66:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
May 01 02:08:54 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:54 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:54.488075 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:54.483871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:44553)
0501 02:08:54.483998 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:54.484007 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:54.484075 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:54.484188 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:54.484189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:54.484348 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:54.484507 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:54.484508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.485200 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:54.485201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:54.485577 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:54.485579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.485644 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:54.486183 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:54.486187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:54.488012 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
W20240501 02:08:54.488241 31820 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:54.483936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:44553 (local address 127.22.12.126:40073)
0501 02:08:54.484016 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:54.484018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:54.484073 (+ 55us) server_negotiation.cc:407] Connection header received
0501 02:08:54.484117 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:54.484118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:54.484131 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:54.484178 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:54.484600 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.485099 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:54.485678 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:54.485800 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:54.488157 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:44553: BlockingRecv error: recv got EOF from 127.22.12.67:44553 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
May 01 02:08:55 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:55 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:55.264333 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:55.260231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:53781)
0501 02:08:55.260361 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:55.260375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:55.260446 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:55.260589 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:55.260590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:55.260753 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:55.260922 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:55.260924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.261534 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:55.261534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:55.261910 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:55.261911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.261962 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:55.262468 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:55.262472 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:55.264267 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
W20240501 02:08:55.264482 31824 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:55.260335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:53781 (local address 127.22.12.126:40073)
0501 02:08:55.260461 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:55.260463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:55.260472 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:55.260496 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:55.260496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:55.260506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:55.260562 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:55.260986 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.261479 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:55.261964 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.262074 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:55.264413 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:53781: BlockingRecv error: recv got EOF from 127.22.12.65:53781 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
May 01 02:08:55 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:55 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:55.377277 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:55.373347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:56191)
0501 02:08:55.373467 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:55.373477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:55.373544 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:55.373683 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:55.373684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:55.373836 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:55.373993 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:55.373994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.374640 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:55.374641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:55.375000 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:55.375001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.375053 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:55.375588 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:55.375593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:55.377219 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20240501 02:08:55.377396 31824 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:55.373420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:56191 (local address 127.22.12.126:40073)
0501 02:08:55.373476 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:55.373478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:55.373527 (+ 49us) server_negotiation.cc:407] Connection header received
0501 02:08:55.373611 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:55.373612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:55.373623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:55.373667 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:55.374056 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.374576 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:55.375062 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.375181 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:55.377318 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:56191: BlockingRecv error: recv got EOF from 127.22.12.66:56191 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
May 01 02:08:55 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:55 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:55.492650 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:55.488549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:43159)
0501 02:08:55.488744 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:55.488755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:55.488829 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:55.488963 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:55.488964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:55.489136 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:55.489304 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:55.489306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.489926 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:55.489927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:55.490292 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:55.490294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.490355 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:55.490914 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:55.490919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:55.492589 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
W20240501 02:08:55.492750 31824 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:55.488631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:43159 (local address 127.22.12.126:40073)
0501 02:08:55.488727 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:55.488730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:55.488826 (+ 96us) server_negotiation.cc:407] Connection header received
0501 02:08:55.488885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:55.488886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:55.488901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:55.488949 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:55.489387 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.489856 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:55.490368 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:55.490560 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:55.492689 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:43159: BlockingRecv error: recv got EOF from 127.22.12.67:43159 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
May 01 02:08:56 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:56 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:56.272691 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:56.264828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:53635)
0501 02:08:56.265055 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:56.265066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:56.265129 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:56.265268 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:56.265269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:56.265432 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:56.265587 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:56.265589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.266204 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:56.266204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:56.266617 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:56.266618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.266673 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:56.270871 (+ 4198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:56.270876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:56.272627 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":164,"threads_started":1}
W20240501 02:08:56.272859 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:56.264956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:53635 (local address 127.22.12.126:40073)
0501 02:08:56.265127 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:56.265129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:56.265165 (+ 36us) server_negotiation.cc:407] Connection header received
0501 02:08:56.265191 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:56.265192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:56.265203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:56.265253 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:56.265647 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.266145 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:56.266709 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.266849 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:56.272788 (+ 5939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:53635: BlockingRecv error: recv got EOF from 127.22.12.65:53635 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":50,"threads_started":1}
May 01 02:08:56 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:56 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:56.381872 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:56.377783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:45885)
0501 02:08:56.377913 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:56.377923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:56.377988 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:56.378112 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:56.378113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:56.378270 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:56.378429 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:56.378430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.379099 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:56.379100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:56.379504 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:56.379506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.379575 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:56.380117 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:56.380121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:56.381811 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
W20240501 02:08:56.381991 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:56.377886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:45885 (local address 127.22.12.126:40073)
0501 02:08:56.377938 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:56.377940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:56.377973 (+ 33us) server_negotiation.cc:407] Connection header received
0501 02:08:56.378048 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:56.378049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:56.378058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:56.378099 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:56.378492 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.378997 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:56.379606 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.379768 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:56.381928 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:45885: BlockingRecv error: recv got EOF from 127.22.12.66:45885 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 01 02:08:56 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:56 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:56.497083 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:56.493185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:36611)
0501 02:08:56.493328 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:56.493338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:56.493411 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:56.493553 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:56.493554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:56.493728 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:56.493881 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:56.493883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.494516 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:56.494518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:56.494940 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:56.494942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.495006 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:56.495489 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:56.495493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:56.497005 (+ 1512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
W20240501 02:08:56.497170 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:56.493331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:36611 (local address 127.22.12.126:40073)
0501 02:08:56.493388 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:56.493390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:56.493399 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:56.493475 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:56.493476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:56.493490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:56.493536 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:56.493965 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.494438 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:56.495023 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:56.495182 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:56.497117 (+ 1935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:36611: BlockingRecv error: recv got EOF from 127.22.12.67:36611 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
May 01 02:08:57 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:57 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:57.277539 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:57.273249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:45479)
0501 02:08:57.273377 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:57.273388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:57.273463 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:57.273661 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:57.273662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:57.273853 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:57.274011 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:57.274014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.274746 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:57.274747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:57.275148 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:57.275150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.275216 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:57.275866 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:57.275870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:57.277477 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
W20240501 02:08:57.277688 31832 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:57.273304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:45479 (local address 127.22.12.126:40073)
0501 02:08:57.273419 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:57.273421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:57.273445 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:08:57.273536 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:57.273537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:57.273549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:57.273621 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:57.274116 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.274646 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:57.275239 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.275367 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:57.277593 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:45479: BlockingRecv error: recv got EOF from 127.22.12.65:45479 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
May 01 02:08:57 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:57 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:57.386677 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:57.382386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:57945)
0501 02:08:57.382572 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:57.382584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:57.382704 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:57.382849 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:57.382850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:57.383051 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:57.383242 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:57.383245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.383907 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:57.383909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:57.384292 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:57.384293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.384357 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:57.384883 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:57.384887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:57.386615 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
W20240501 02:08:57.386811 31832 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:57.382495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:57945 (local address 127.22.12.126:40073)
0501 02:08:57.382581 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:57.382583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:57.382691 (+ 108us) server_negotiation.cc:407] Connection header received
0501 02:08:57.382771 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:57.382772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:57.382784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:57.382834 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:57.383320 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.383825 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:57.384362 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.384520 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:57.386736 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:57945: BlockingRecv error: recv got EOF from 127.22.12.66:57945 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
May 01 02:08:57 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:57 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:57.501744 31834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:57.497645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:38455)
0501 02:08:57.497793 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:57.497802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:57.497865 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:57.498001 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:57.498002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:57.498173 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:57.498333 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:57.498335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.498996 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:57.498998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:57.499390 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:57.499392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.499467 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:57.499942 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:57.499946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:57.501657 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
W20240501 02:08:57.501793 31832 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:57.497721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:38455 (local address 127.22.12.126:40073)
0501 02:08:57.497776 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:57.497778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:57.497859 (+ 81us) server_negotiation.cc:407] Connection header received
0501 02:08:57.497931 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:57.497931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:57.497944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:57.497984 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:57.498394 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.498893 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:57.499460 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:57.499561 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:57.501732 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:38455: BlockingRecv error: recv got EOF from 127.22.12.67:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
May 01 02:08:58 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:58 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:58.282374 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:58.278107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:60227)
0501 02:08:58.278258 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:58.278268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:58.278322 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:58.278506 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:58.278507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:58.278761 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:58.278942 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:58.278945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.279712 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:58.279712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:58.280121 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:58.280123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.280184 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:58.280706 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:58.280710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:58.282307 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20240501 02:08:58.282483 31836 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:58.278217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:60227 (local address 127.22.12.126:40073)
0501 02:08:58.278340 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:58.278341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:58.278350 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:58.278375 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:58.278376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:58.278385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:58.278454 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:58.279089 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.279637 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:58.280217 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.280358 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:58.282417 (+ 2059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:60227: BlockingRecv error: recv got EOF from 127.22.12.65:60227 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
May 01 02:08:58 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:58 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:58.391232 31837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:58.387146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:56287)
0501 02:08:58.387267 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:58.387277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:58.387358 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:58.387518 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:58.387519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:58.387704 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:58.387859 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:58.387861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.388523 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:58.388524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:58.388914 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:58.388916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.388977 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:58.389484 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:58.389487 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:58.391154 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
W20240501 02:08:58.391319 31836 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:58.387246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:56287 (local address 127.22.12.126:40073)
0501 02:08:58.387306 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:58.387309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:58.387324 (+ 15us) server_negotiation.cc:407] Connection header received
0501 02:08:58.387420 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:58.387420 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:58.387454 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:58.387529 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:58.387938 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.388430 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:58.388986 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.389151 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:58.391264 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:56287: BlockingRecv error: recv got EOF from 127.22.12.66:56287 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
May 01 02:08:58 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:58 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:58.506498 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:58.502310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:34131)
0501 02:08:58.502487 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:58.502498 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:58.502588 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:58.502730 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:58.502731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:58.502951 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:58.503117 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:58.503119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.503794 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:58.503795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:58.504208 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:58.504210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.504276 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:58.504772 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:58.504776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:58.506439 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
W20240501 02:08:58.506668 31836 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:58.502414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:34131 (local address 127.22.12.126:40073)
0501 02:08:58.502465 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:58.502467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:58.502583 (+ 116us) server_negotiation.cc:407] Connection header received
0501 02:08:58.502639 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:58.502639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:58.502657 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:58.502713 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:58.503181 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.503672 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:58.504276 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:58.504393 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:58.506596 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:34131: BlockingRecv error: recv got EOF from 127.22.12.67:34131 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
May 01 02:08:59 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:59 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:59.287118 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:59.282889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:42361)
0501 02:08:59.283010 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:59.283024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:59.283089 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:59.283221 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:59.283222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:59.283382 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:59.283556 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:59.283558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.284245 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:59.284246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:59.284660 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:59.284661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.284729 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:59.285214 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:59.285218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:59.287056 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
W20240501 02:08:59.287258 31840 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:59.282948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:42361 (local address 127.22.12.126:40073)
0501 02:08:59.283076 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:59.283078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:59.283087 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:08:59.283141 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:59.283141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:59.283152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:59.283210 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:59.283632 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.284142 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:59.284738 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.284868 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:59.287170 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:42361: BlockingRecv error: recv got EOF from 127.22.12.65:42361 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
May 01 02:08:59 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:59 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:59.395597 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:59.391728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:34391)
0501 02:08:59.391894 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:59.391907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:59.391974 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:59.392099 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:59.392099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:59.392279 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:59.392434 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:59.392435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.393053 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:59.393054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:59.393414 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:59.393415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.393468 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:59.393934 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:59.393938 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:59.395537 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
W20240501 02:08:59.395757 31840 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:59.391872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:34391 (local address 127.22.12.126:40073)
0501 02:08:59.391929 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:59.391931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:59.391957 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:08:59.392028 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:59.392029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:59.392039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:59.392088 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:59.392502 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.392990 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:59.393471 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.393579 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:59.395680 (+ 2101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:34391: BlockingRecv error: recv got EOF from 127.22.12.66:34391 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
May 01 02:08:59 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:08:59 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:08:59.510882 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:08:59.507029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:50057)
0501 02:08:59.507180 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:59.507193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:08:59.507280 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:08:59.507416 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:08:59.507417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:08:59.507588 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:08:59.507740 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:59.507742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.508382 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:59.508383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:08:59.508752 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:08:59.508753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.508812 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:59.509285 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:08:59.509289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:08:59.510807 (+ 1518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
W20240501 02:08:59.510982 31840 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:08:59.507111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:50057 (local address 127.22.12.126:40073)
0501 02:08:59.507208 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0501 02:08:59.507211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:08:59.507271 (+ 60us) server_negotiation.cc:407] Connection header received
0501 02:08:59.507343 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:08:59.507344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:08:59.507358 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:08:59.507404 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:08:59.507817 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.508306 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:59.508835 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:59.508991 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:59.510927 (+ 1936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:50057: BlockingRecv error: recv got EOF from 127.22.12.67:50057 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
May 01 02:09:00 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:09:00 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:09:00.291775 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:00.287576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:48961)
0501 02:09:00.287709 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:00.287719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:00.287808 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:00.287976 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:00.287977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:00.288133 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:00.288287 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:00.288289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.288985 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:00.288986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:00.289381 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:00.289383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.289449 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:00.289955 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:00.289960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:00.291693 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20240501 02:09:00.291877 31844 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:00.287677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:48961 (local address 127.22.12.126:40073)
0501 02:09:00.287854 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:00.287856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:00.287865 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:09:00.287886 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:00.287887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:00.287897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:00.287969 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:00.288343 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.288890 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:00.289453 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.289573 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:00.291806 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:48961: BlockingRecv error: recv got EOF from 127.22.12.65:48961 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
May 01 02:09:00 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:09:00 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:09:00.399961 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:00.396057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:38371)
0501 02:09:00.396189 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:00.396199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:00.396262 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:00.396388 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:00.396389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:00.396552 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:00.396709 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:00.396711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.397318 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:00.397319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:00.397692 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:00.397693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.397756 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:00.398251 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:00.398255 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:00.399900 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
W20240501 02:09:00.400067 31844 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:00.396130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:38371 (local address 127.22.12.126:40073)
0501 02:09:00.396195 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:00.396197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:00.396259 (+ 62us) server_negotiation.cc:407] Connection header received
0501 02:09:00.396322 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:00.396322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:00.396333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:00.396379 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:00.396775 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.397253 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:00.397754 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.397859 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:00.400007 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:38371: BlockingRecv error: recv got EOF from 127.22.12.66:38371 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
May 01 02:09:00 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:09:00 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.67@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:09:00.515481 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:00.511411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.67:40055)
0501 02:09:00.511535 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:00.511552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:00.511630 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:00.511764 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:00.511765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:00.511942 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:00.512095 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:00.512097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.512774 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:00.512775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:00.513225 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:00.513227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.513288 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:00.513775 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:00.513779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:00.515422 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20240501 02:09:00.515594 31844 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:00.511529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:40055 (local address 127.22.12.126:40073)
0501 02:09:00.511602 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:00.511604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:00.511615 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:09:00.511694 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:00.511695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:00.511707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:00.511750 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:00.512179 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.512694 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:00.513303 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:00.513436 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:00.515524 (+ 2088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.67:40055: BlockingRecv error: recv got EOF from 127.22.12.67:40055 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
May 01 02:09:01 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:09:01 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.65@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:09:01.301458 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.292324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.65:33443)
0501 02:09:01.292469 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:01.292482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:01.292552 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:01.292679 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:01.292679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:01.292861 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:01.293026 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.293028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.293705 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.293706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:01.294114 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.294116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.294190 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.299674 (+ 5484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:01.299680 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:01.301396 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20240501 02:09:01.301597 31848 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:01.292419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:33443 (local address 127.22.12.126:40073)
0501 02:09:01.292545 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:01.292547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:01.292557 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:09:01.292608 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:01.292609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:01.292619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:01.292664 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:01.293089 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.293598 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.294191 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.294308 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.301525 (+ 7217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.65:33443: BlockingRecv error: recv got EOF from 127.22.12.65:33443 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
May 01 02:09:01 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
May 01 02:09:01 dist-test-slave-9tkh krb5kdc[30790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.12.66@KRBTEST.COM for oryx/127.22.12.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 02:09:01.404528 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.400501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40073 (local address 127.22.12.66:58023)
0501 02:09:01.400622 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:01.400633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:01.400703 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:01.400842 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:01.400843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:01.401016 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:01.401169 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.401172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.401825 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.401826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:01.402188 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.402189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.402252 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.402826 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:01.402830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:01.404468 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.12.126:40073: Server oryx/127.22.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20240501 02:09:01.404664 31848 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:01.400597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:58023 (local address 127.22.12.126:40073)
0501 02:09:01.400680 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:01.400682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:01.400691 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:09:01.400768 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:01.400769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:01.400781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:01.400829 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:01.401255 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.401743 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.402264 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.402422 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.404598 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.12.66:58023: BlockingRecv error: recv got EOF from 127.22.12.66:58023 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20240501 02:09:01.450124 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31373
I20240501 02:09:01.454406 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31525
I20240501 02:09:01.458281 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31657
I20240501 02:09:01.462157 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31303
2024-05-01T02:09:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17189 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2024-05-01T02:09:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:09:01Z Disabled control of system clock
I20240501 02:09:01.488672 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:46397
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:46397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240501 02:09:01.567127 31853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:01.567319 31853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:01.567348 31853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:01.568656 31853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:09:01.568699 31853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:01.568715 31853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:01.568732 31853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:09:01.568745 31853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:09:01.570179 31853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:46397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:01.570425 31853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:01.570673 31853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:01.576066 31858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.576347 31859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.576709 31861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:01.576733 31853 server_base.cc:1034] running on GCE node
I20240501 02:09:01.576995 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:01.577236 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:01.578363 31853 hybrid_clock.cc:648] HybridClock initialized: now 1714529341578326 us; error 54 us; skew 500 ppm
I20240501 02:09:01.579687 31853 webserver.cc:460] Webserver started at http://127.22.12.126:37049/ using document root <none> and password file <none>
I20240501 02:09:01.579890 31853 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:01.579931 31853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:01.580035 31853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:01.581032 31853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "34b127c8b500499d955e6b2cfbb8aec5"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "cc77507b3395bc3b1d54268f9a6e3a9f"
server_key_iv: "32b3221f4f11108f281b76d6a82fb522"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.581310 31853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "34b127c8b500499d955e6b2cfbb8aec5"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "cc77507b3395bc3b1d54268f9a6e3a9f"
server_key_iv: "32b3221f4f11108f281b76d6a82fb522"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.582409 31853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:09:01.583146 31867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.583335 31853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:09:01.583413 31853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "34b127c8b500499d955e6b2cfbb8aec5"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "cc77507b3395bc3b1d54268f9a6e3a9f"
server_key_iv: "32b3221f4f11108f281b76d6a82fb522"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.583472 31853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:01.592725 31853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:01.593006 31853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:01.593107 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:01.596719 31853 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:46397
I20240501 02:09:01.596756 31919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:46397 every 8 connection(s)
I20240501 02:09:01.597056 31853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:09:01.597649 31920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:01.600284 31920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5: Bootstrap starting.
I20240501 02:09:01.600884 31920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:01.601125 31920 log.cc:824] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:01.601805 31920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5: No bootstrap required, opened a new log
I20240501 02:09:01.602519 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31853
I20240501 02:09:01.602654 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:09:01.602842 22577 external_mini_cluster.cc:1428] Setting key e65d7a5119bf9611377e0ca5b04410b5
I20240501 02:09:01.603749 31920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b127c8b500499d955e6b2cfbb8aec5" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46397 } }
I20240501 02:09:01.603902 31920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:01.603928 31920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b127c8b500499d955e6b2cfbb8aec5, State: Initialized, Role: FOLLOWER
I20240501 02:09:01.604079 31920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b127c8b500499d955e6b2cfbb8aec5" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46397 } }
I20240501 02:09:01.604157 31920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:09:01.604188 31920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:09:01.604234 31920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:01.604797 31920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b127c8b500499d955e6b2cfbb8aec5" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46397 } }
I20240501 02:09:01.604883 31920 leader_election.cc:302] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b127c8b500499d955e6b2cfbb8aec5; no voters:
I20240501 02:09:01.605048 31920 leader_election.cc:288] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:09:01.605262 31925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:09:01.605305 31920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:09:01.605469 31925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [term 1 LEADER]: Becoming Leader. State: Replica: 34b127c8b500499d955e6b2cfbb8aec5, State: Running, Role: LEADER
I20240501 02:09:01.605655 31925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b127c8b500499d955e6b2cfbb8aec5" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46397 } }
I20240501 02:09:01.605993 31926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b127c8b500499d955e6b2cfbb8aec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b127c8b500499d955e6b2cfbb8aec5" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46397 } } }
I20240501 02:09:01.606062 31926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:01.606269 31927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b127c8b500499d955e6b2cfbb8aec5. Latest consensus state: current_term: 1 leader_uuid: "34b127c8b500499d955e6b2cfbb8aec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b127c8b500499d955e6b2cfbb8aec5" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 46397 } } }
I20240501 02:09:01.606508 31927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5 [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:01.606266 31934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:09:01.607420 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.603360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.22.12.126:46397)
0501 02:09:01.603591 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:01.603600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:01.603622 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:09:01.603749 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:01.603755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:01.603825 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:01.603981 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:01.604354 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.605215 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.605904 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.606504 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.606841 (+ 337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:01.606849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:01.606850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:01.607014 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:01.607039 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:01.607046 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:01.607105 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:01.607207 (+ 102us) server_negotiation.cc:299] Negotiation successful
0501 02:09:01.607273 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20240501 02:09:01.607721 31934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:09:01.609104 31934 catalog_manager.cc:1347] Generated new cluster ID: 3a9dd615d35e452886af4c87695386e1
I20240501 02:09:01.609163 31934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:09:01.627367 31934 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:09:01.627923 31934 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:09:01.634682 31934 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 34b127c8b500499d955e6b2cfbb8aec5: Generated new TSK 0
I20240501 02:09:01.634953 31934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:09:01.637871 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46397
--builtin_ntp_servers=127.22.12.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240501 02:09:01.717880 31944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:01.718060 31944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:01.718081 31944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:01.719514 31944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:01.719566 31944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:01.719590 31944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:09:01.721072 31944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:01.721295 31944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:01.721539 31944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:01.726214 31950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.726591 31949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.727015 31952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:01.727571 31944 server_base.cc:1034] running on GCE node
I20240501 02:09:01.727764 31944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:01.728017 31944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:01.729224 31944 hybrid_clock.cc:648] HybridClock initialized: now 1714529341729202 us; error 41 us; skew 500 ppm
I20240501 02:09:01.730763 31944 webserver.cc:460] Webserver started at http://127.22.12.65:40931/ using document root <none> and password file <none>
I20240501 02:09:01.730978 31944 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:01.731021 31944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:01.731125 31944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:01.732144 31944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "0783e283c46c4feca69368b884a3255b"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "56c58ad2af4b6f94323a0858a161ed72"
server_key_iv: "8477e97d80ec51c662a24cf074078d2c"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.732432 31944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "0783e283c46c4feca69368b884a3255b"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "56c58ad2af4b6f94323a0858a161ed72"
server_key_iv: "8477e97d80ec51c662a24cf074078d2c"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.733570 31944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:09:01.734248 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.734429 31944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:09:01.734505 31944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "0783e283c46c4feca69368b884a3255b"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "56c58ad2af4b6f94323a0858a161ed72"
server_key_iv: "8477e97d80ec51c662a24cf074078d2c"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.734601 31944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:01.747913 31944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:01.748220 31944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:01.748332 31944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:01.748564 31944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:01.748889 31944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:01.748924 31944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.748953 31944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:01.748968 31944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.755399 31944 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:44503
I20240501 02:09:01.755486 32071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:44503 every 8 connection(s)
I20240501 02:09:01.755756 31944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:09:01.760401 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.756444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:60149 (local address 127.22.12.126:46397)
0501 02:09:01.756502 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:01.756504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:01.756711 (+ 207us) server_negotiation.cc:407] Connection header received
0501 02:09:01.757303 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:01.757304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:01.757315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:01.757365 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:01.757906 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.758376 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.759146 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.759272 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.760176 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:01.760180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:01.760180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:01.760184 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:01.760198 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:01.760201 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:01.760241 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:01.760306 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 02:09:01.760336 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:09:01.760496 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.756384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:46397 (local address 127.22.12.65:60149)
0501 02:09:01.756608 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:01.756636 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:01.757193 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:01.757424 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:01.757428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:01.757437 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:01.757825 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.757828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.758453 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.758454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:01.759066 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.759069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.759196 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.760070 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:09:01.760070 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:01.760087 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:09:01.760104 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:01.760248 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:01.760251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:01.760253 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:09:01.760304 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:09:01.760360 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20240501 02:09:01.760948 32072 heartbeater.cc:347] Connected to a master server at 127.22.12.126:46397
I20240501 02:09:01.761029 32072 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:01.761200 32072 heartbeater.cc:511] Master 127.22.12.126:46397 requested a full tablet report, sending...
I20240501 02:09:01.761644 31884 ts_manager.cc:194] Registered new tserver with Master: 0783e283c46c4feca69368b884a3255b (127.22.12.65:44503)
I20240501 02:09:01.762243 31884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.65:60149
I20240501 02:09:01.763316 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 31944
I20240501 02:09:01.763406 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:09:01.763537 22577 external_mini_cluster.cc:1428] Setting key 7cefa0f8856145be181022728b4bc758
I20240501 02:09:01.764369 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46397
--builtin_ntp_servers=127.22.12.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240501 02:09:01.843289 32075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:01.843480 32075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:01.843503 32075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:01.844930 32075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:01.844992 32075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:01.845018 32075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:09:01.846634 32075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:01.846884 32075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:01.847123 32075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:01.851514 32080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.851843 32083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.852283 32081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:01.852725 32075 server_base.cc:1034] running on GCE node
I20240501 02:09:01.852892 32075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:01.853080 32075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:01.854214 32075 hybrid_clock.cc:648] HybridClock initialized: now 1714529341854203 us; error 29 us; skew 500 ppm
I20240501 02:09:01.855461 32075 webserver.cc:460] Webserver started at http://127.22.12.66:41969/ using document root <none> and password file <none>
I20240501 02:09:01.855633 32075 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:01.855672 32075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:01.855778 32075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:01.856915 32075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "fa1c32961efd48ba870f1b749f114817"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "8da0ef0a0c6062cb1a0087e3fdb7d024"
server_key_iv: "70a689ddde85fbde3f7b96ccbe16b102"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.857201 32075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "fa1c32961efd48ba870f1b749f114817"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "8da0ef0a0c6062cb1a0087e3fdb7d024"
server_key_iv: "70a689ddde85fbde3f7b96ccbe16b102"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.858251 32075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:09:01.858912 32089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.859067 32075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.859153 32075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "fa1c32961efd48ba870f1b749f114817"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "8da0ef0a0c6062cb1a0087e3fdb7d024"
server_key_iv: "70a689ddde85fbde3f7b96ccbe16b102"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.859232 32075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:01.876703 32075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:01.877002 32075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:01.877115 32075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:01.877339 32075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:01.877676 32075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:01.877710 32075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.877739 32075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:01.877756 32075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.882956 32075 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:43517
I20240501 02:09:01.882997 32202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:43517 every 8 connection(s)
I20240501 02:09:01.883297 32075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:09:01.887929 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.883973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:50893 (local address 127.22.12.126:46397)
0501 02:09:01.884034 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:01.884036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:01.884217 (+ 181us) server_negotiation.cc:407] Connection header received
0501 02:09:01.884662 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:01.884663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:01.884673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:01.884732 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:01.885148 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.885637 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.886398 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.886596 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.887604 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:01.887609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:01.887609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:01.887615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:01.887632 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:01.887636 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:01.887686 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:01.887811 (+ 125us) server_negotiation.cc:299] Negotiation successful
0501 02:09:01.887864 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:09:01.888118 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:01.883944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:46397 (local address 127.22.12.66:50893)
0501 02:09:01.884120 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:01.884149 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:01.884580 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:01.884778 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:01.884781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:01.884790 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:01.885089 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.885092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.885758 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:01.885759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:01.886285 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:01.886289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:01.886404 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:01.887445 (+ 1041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:09:01.887446 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:01.887467 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:09:01.887488 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:01.887716 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:01.887722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:01.887724 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:09:01.887807 (+ 83us) client_negotiation.cc:240] Negotiation successful
0501 02:09:01.887923 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20240501 02:09:01.888666 32203 heartbeater.cc:347] Connected to a master server at 127.22.12.126:46397
I20240501 02:09:01.888692 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 32075
I20240501 02:09:01.888761 32203 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:01.888772 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:09:01.888932 22577 external_mini_cluster.cc:1428] Setting key a78ac520264a48e1302aadc9d79dfa0e
I20240501 02:09:01.888983 32203 heartbeater.cc:511] Master 127.22.12.126:46397 requested a full tablet report, sending...
I20240501 02:09:01.889499 31883 ts_manager.cc:194] Registered new tserver with Master: fa1c32961efd48ba870f1b749f114817 (127.22.12.66:43517)
I20240501 02:09:01.889895 31883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.66:50893
I20240501 02:09:01.890043 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46397
--builtin_ntp_servers=127.22.12.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240501 02:09:01.966667 32206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:01.966840 32206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:01.966867 32206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:01.968163 32206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:01.968209 32206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:01.968240 32206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:09:01.969642 32206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:46397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:01.969864 32206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:01.970090 32206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:01.974398 32212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.974800 32214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:01.975203 32211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:01.975662 32206 server_base.cc:1034] running on GCE node
I20240501 02:09:01.975811 32206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:01.975989 32206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:01.977123 32206 hybrid_clock.cc:648] HybridClock initialized: now 1714529341977105 us; error 33 us; skew 500 ppm
I20240501 02:09:01.978325 32206 webserver.cc:460] Webserver started at http://127.22.12.67:34485/ using document root <none> and password file <none>
I20240501 02:09:01.978508 32206 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:01.978586 32206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:01.978703 32206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:01.979694 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "b3138d25b557412b9da64b4d149f35ba"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "eccf08889f3b9d0ec462adf5d95c1671"
server_key_iv: "c4e239532a294e5d0dfeeafb1d46737a"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.979992 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "b3138d25b557412b9da64b4d149f35ba"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "eccf08889f3b9d0ec462adf5d95c1671"
server_key_iv: "c4e239532a294e5d0dfeeafb1d46737a"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.981072 32206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:01.981699 32220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.981870 32206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:09:01.981947 32206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "b3138d25b557412b9da64b4d149f35ba"
format_stamp: "Formatted at 2024-05-01 02:09:01 on dist-test-slave-9tkh"
server_key: "eccf08889f3b9d0ec462adf5d95c1671"
server_key_iv: "c4e239532a294e5d0dfeeafb1d46737a"
server_key_version: "encryptionkey@0"
I20240501 02:09:01.982003 32206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:01.997121 32206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:01.997406 32206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:01.997505 32206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:01.997704 32206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:01.998008 32206 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:01.998040 32206 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:01.998068 32206 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:01.998097 32206 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.003298 32206 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:45239
I20240501 02:09:02.003340 32333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:45239 every 8 connection(s)
I20240501 02:09:02.003636 32206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:09:02.004357 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 32206
I20240501 02:09:02.004462 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:09:02.004642 22577 external_mini_cluster.cc:1428] Setting key c6e522a2b511b724ee4887dff3763c5b
I20240501 02:09:02.008109 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.004322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:39189 (local address 127.22.12.126:46397)
0501 02:09:02.004391 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.004394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.004606 (+ 212us) server_negotiation.cc:407] Connection header received
0501 02:09:02.005219 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.005220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.005231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.005284 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.005779 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.006356 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.006917 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.007039 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.007887 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.007891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:02.007891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:02.007895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:02.007910 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:02.007914 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.007955 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:02.008021 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 02:09:02.008049 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:09:02.008213 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.004288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:46397 (local address 127.22.12.67:39189)
0501 02:09:02.004500 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.004530 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:02.005119 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:02.005337 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:02.005340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:02.005349 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:02.005720 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:02.005724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.006423 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.006424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:02.006836 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:02.006838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.006961 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.007781 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:09:02.007781 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:02.007798 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:09:02.007816 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:02.007963 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:02.007965 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:02.007967 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:09:02.008019 (+ 52us) client_negotiation.cc:240] Negotiation successful
0501 02:09:02.008073 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20240501 02:09:02.008615 32334 heartbeater.cc:347] Connected to a master server at 127.22.12.126:46397
I20240501 02:09:02.008704 32334 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:02.008915 32334 heartbeater.cc:511] Master 127.22.12.126:46397 requested a full tablet report, sending...
I20240501 02:09:02.009270 31883 ts_manager.cc:194] Registered new tserver with Master: b3138d25b557412b9da64b4d149f35ba (127.22.12.67:45239)
I20240501 02:09:02.009624 31883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.67:39189
I20240501 02:09:02.016292 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240501 02:09:02.018007 31924 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:02.017566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.22.12.126:46397)
0501 02:09:02.017630 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.017632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.017698 (+ 66us) server_negotiation.cc:407] Connection header received
0501 02:09:02.017751 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.017752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.017762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.017809 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.017949 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49184: BlockingRecv error: recv got EOF from 127.0.0.1:49184 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 02:09:02.018069 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31944
I20240501 02:09:02.022179 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 32075
I20240501 02:09:02.026295 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 32206
I20240501 02:09:02.029985 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 31853
2024-05-01T02:09:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (566 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2024-05-01T02:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:09:02Z Disabled control of system clock
I20240501 02:09:02.054062 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35787
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240501 02:09:02.131062 32345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:02.131259 32345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:02.131279 32345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:02.132594 32345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:09:02.132635 32345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:02.132647 32345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:02.132658 32345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:09:02.132668 32345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:09:02.134095 32345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:02.134289 32345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:02.134490 32345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:02.138512 32353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.138950 32350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.139377 32351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:02.139827 32345 server_base.cc:1034] running on GCE node
I20240501 02:09:02.140038 32345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:02.140251 32345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:02.141384 32345 hybrid_clock.cc:648] HybridClock initialized: now 1714529342141371 us; error 28 us; skew 500 ppm
I20240501 02:09:02.142494 32345 webserver.cc:460] Webserver started at http://127.22.12.126:35139/ using document root <none> and password file <none>
I20240501 02:09:02.142694 32345 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:02.142738 32345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:02.142843 32345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:02.143867 32345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "46af4d36c88a4ef797ac586f82116762"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "22133c0c095ecd986c07ef1ec2e56114"
server_key_iv: "5e715cd28e940e844d17285ae1279fbd"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.144153 32345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "46af4d36c88a4ef797ac586f82116762"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "22133c0c095ecd986c07ef1ec2e56114"
server_key_iv: "5e715cd28e940e844d17285ae1279fbd"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.145238 32345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:09:02.145874 32359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.146027 32345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:09:02.146103 32345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "46af4d36c88a4ef797ac586f82116762"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "22133c0c095ecd986c07ef1ec2e56114"
server_key_iv: "5e715cd28e940e844d17285ae1279fbd"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.146162 32345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:02.149724 32345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:02.149995 32345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:02.150089 32345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:02.153599 32345 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:35787
I20240501 02:09:02.153626 32411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:35787 every 8 connection(s)
I20240501 02:09:02.153908 32345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:09:02.154459 32412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:02.156791 32412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762: Bootstrap starting.
I20240501 02:09:02.157289 32412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:02.157531 32412 log.cc:824] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:02.157615 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 32345
I20240501 02:09:02.157703 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:09:02.157847 22577 external_mini_cluster.cc:1428] Setting key 083916262374e7b2462dc534e8cf4b3e
I20240501 02:09:02.158169 32412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762: No bootstrap required, opened a new log
I20240501 02:09:02.160025 32412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46af4d36c88a4ef797ac586f82116762" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35787 } }
I20240501 02:09:02.160159 32412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:02.160183 32412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46af4d36c88a4ef797ac586f82116762, State: Initialized, Role: FOLLOWER
I20240501 02:09:02.160280 32412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46af4d36c88a4ef797ac586f82116762" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35787 } }
I20240501 02:09:02.160354 32412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:09:02.160398 32412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:09:02.160446 32412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:02.160657 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.158325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.22.12.126:35787)
0501 02:09:02.158522 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.158559 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.158583 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:09:02.159256 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.159266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.159310 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.159472 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.160206 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.160209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:02.160209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:02.160213 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:02.160252 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:02.160259 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.160330 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:02.160477 (+ 147us) server_negotiation.cc:299] Negotiation successful
0501 02:09:02.160528 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20240501 02:09:02.161052 32412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46af4d36c88a4ef797ac586f82116762" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35787 } }
I20240501 02:09:02.161141 32412 leader_election.cc:302] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46af4d36c88a4ef797ac586f82116762; no voters:
I20240501 02:09:02.161296 32412 leader_election.cc:288] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:09:02.161329 32417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:09:02.161491 32417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [term 1 LEADER]: Becoming Leader. State: Replica: 46af4d36c88a4ef797ac586f82116762, State: Running, Role: LEADER
I20240501 02:09:02.161518 32412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:09:02.161573 32417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46af4d36c88a4ef797ac586f82116762" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35787 } }
I20240501 02:09:02.161923 32419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46af4d36c88a4ef797ac586f82116762. Latest consensus state: current_term: 1 leader_uuid: "46af4d36c88a4ef797ac586f82116762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46af4d36c88a4ef797ac586f82116762" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35787 } } }
I20240501 02:09:02.161998 32419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:02.161906 32418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46af4d36c88a4ef797ac586f82116762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46af4d36c88a4ef797ac586f82116762" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35787 } } }
I20240501 02:09:02.162133 32418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762 [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:02.162238 32421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:09:02.162689 32421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:09:02.163935 32421 catalog_manager.cc:1347] Generated new cluster ID: 54f88bab6cb04c02b5cce99a41c7c4dd
I20240501 02:09:02.163982 32421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:09:02.188797 32421 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:09:02.188978 32421 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:09:02.194505 32421 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 46af4d36c88a4ef797ac586f82116762: Generated new TSK 0
I20240501 02:09:02.194725 32421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:09:02.197904 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35787
--builtin_ntp_servers=127.22.12.84:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240501 02:09:02.276499 32436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:02.276700 32436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:02.276724 32436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:02.278092 32436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:02.278151 32436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:02.278180 32436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:09:02.279764 32436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:02.279990 32436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:02.280232 32436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:02.285324 32444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.285651 32441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.286093 32442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:02.286567 32436 server_base.cc:1034] running on GCE node
I20240501 02:09:02.286746 32436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:02.286964 32436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:02.288115 32436 hybrid_clock.cc:648] HybridClock initialized: now 1714529342288093 us; error 38 us; skew 500 ppm
I20240501 02:09:02.289429 32436 webserver.cc:460] Webserver started at http://127.22.12.65:34147/ using document root <none> and password file <none>
I20240501 02:09:02.289614 32436 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:02.289652 32436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:02.289746 32436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:02.290845 32436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "6cc9e3d563b74f2eabe1953fb65d65b4"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "3da57e2441909b837b3ed1c68576e65e"
server_key_iv: "3a005adf7d9cfb1b7520aebc16837de4"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.291131 32436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "6cc9e3d563b74f2eabe1953fb65d65b4"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "3da57e2441909b837b3ed1c68576e65e"
server_key_iv: "3a005adf7d9cfb1b7520aebc16837de4"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.292315 32436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:02.293011 32450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.293195 32436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:09:02.293272 32436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "6cc9e3d563b74f2eabe1953fb65d65b4"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "3da57e2441909b837b3ed1c68576e65e"
server_key_iv: "3a005adf7d9cfb1b7520aebc16837de4"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.293334 32436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:02.296959 32436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:02.297200 32436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:02.297297 32436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:02.297483 32436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:02.297784 32436 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:02.297818 32436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.297853 32436 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:02.297878 32436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.304266 32436 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:40185
I20240501 02:09:02.304288 32563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:40185 every 8 connection(s)
I20240501 02:09:02.304620 32436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:09:02.307029 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.305293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:52965 (local address 127.22.12.126:35787)
0501 02:09:02.305345 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.305347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.305550 (+ 203us) server_negotiation.cc:407] Connection header received
0501 02:09:02.305984 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.305985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.305990 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.306069 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.306800 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.306801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:02.306801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:02.306806 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:02.306822 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:02.306826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.306884 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:02.306945 (+ 61us) server_negotiation.cc:299] Negotiation successful
0501 02:09:02.306959 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:09:02.307147 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.305265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35787 (local address 127.22.12.65:52965)
0501 02:09:02.305448 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.305477 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:02.305900 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:02.306114 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:02.306117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:02.306125 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:02.306676 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:09:02.306676 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:02.306700 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:09:02.306724 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:02.306885 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:02.306888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:02.306890 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:09:02.306957 (+ 67us) client_negotiation.cc:240] Negotiation successful
0501 02:09:02.306996 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20240501 02:09:02.307929 32564 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35787
I20240501 02:09:02.308013 32564 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:02.308136 32564 heartbeater.cc:511] Master 127.22.12.126:35787 requested a full tablet report, sending...
I20240501 02:09:02.308602 32376 ts_manager.cc:194] Registered new tserver with Master: 6cc9e3d563b74f2eabe1953fb65d65b4 (127.22.12.65:40185)
I20240501 02:09:02.312397 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 32436
I20240501 02:09:02.312479 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:09:02.312649 22577 external_mini_cluster.cc:1428] Setting key 178f540e6bbab1a95114fbecaf5ccc74
I20240501 02:09:02.313477 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35787
--builtin_ntp_servers=127.22.12.84:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240501 02:09:02.391638 32567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:02.391845 32567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:02.391868 32567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:02.393260 32567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:02.393314 32567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:02.393340 32567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:09:02.394891 32567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:02.395125 32567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:02.395370 32567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:02.399785 32573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.400221 32575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.400588 32572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:02.401103 32567 server_base.cc:1034] running on GCE node
I20240501 02:09:02.401289 32567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:02.401487 32567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:02.402601 32567 hybrid_clock.cc:648] HybridClock initialized: now 1714529342402586 us; error 32 us; skew 500 ppm
I20240501 02:09:02.403892 32567 webserver.cc:460] Webserver started at http://127.22.12.66:38885/ using document root <none> and password file <none>
I20240501 02:09:02.404074 32567 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:02.404114 32567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:02.404217 32567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:02.405184 32567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "b085706f13ad4dc49dff89a15e926463"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "34b8781692e37378eb67efcbdcc2e10c"
server_key_iv: "94da6a8dd4e366d372ab9bc6ab97fd68"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.405467 32567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "b085706f13ad4dc49dff89a15e926463"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "34b8781692e37378eb67efcbdcc2e10c"
server_key_iv: "94da6a8dd4e366d372ab9bc6ab97fd68"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.406574 32567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:02.407253 32581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.407404 32567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:09:02.407478 32567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "b085706f13ad4dc49dff89a15e926463"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "34b8781692e37378eb67efcbdcc2e10c"
server_key_iv: "94da6a8dd4e366d372ab9bc6ab97fd68"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.407545 32567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:02.411098 32567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:02.411304 32567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:02.411397 32567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:02.411554 32567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:02.411811 32567 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:02.411841 32567 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.411867 32567 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:02.411890 32567 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.418291 32567 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:43955
I20240501 02:09:02.418339 32694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:43955 every 8 connection(s)
I20240501 02:09:02.418722 32567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:09:02.421209 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.419417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:58419 (local address 127.22.12.126:35787)
0501 02:09:02.419469 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.419471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.419680 (+ 209us) server_negotiation.cc:407] Connection header received
0501 02:09:02.420130 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.420131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.420137 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.420221 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.420962 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.420964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:02.420964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:02.420969 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:02.420987 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:02.420991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.421053 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:02.421115 (+ 62us) server_negotiation.cc:299] Negotiation successful
0501 02:09:02.421130 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:09:02.421296 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.419393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35787 (local address 127.22.12.66:58419)
0501 02:09:02.419572 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.419601 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:02.420038 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:02.420268 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:02.420271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:02.420279 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:02.420838 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:09:02.420838 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:02.420857 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:09:02.420887 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:02.421050 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:02.421052 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:02.421054 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:09:02.421123 (+ 69us) client_negotiation.cc:240] Negotiation successful
0501 02:09:02.421161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20240501 02:09:02.422065 32695 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35787
I20240501 02:09:02.422150 32695 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:02.422276 32695 heartbeater.cc:511] Master 127.22.12.126:35787 requested a full tablet report, sending...
I20240501 02:09:02.422644 32376 ts_manager.cc:194] Registered new tserver with Master: b085706f13ad4dc49dff89a15e926463 (127.22.12.66:43955)
I20240501 02:09:02.427520 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 32567
I20240501 02:09:02.427608 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:09:02.427750 22577 external_mini_cluster.cc:1428] Setting key 1e92523cb8c95952c14dc5e1f6e8cb26
I20240501 02:09:02.428670 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35787
--builtin_ntp_servers=127.22.12.84:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240501 02:09:02.510673 32698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:02.510864 32698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:02.510893 32698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:02.512240 32698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:02.512297 32698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:02.512329 32698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:09:02.513821 32698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:02.514055 32698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:02.514286 32698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:02.518653 32704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.518996 32706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:02.519438 32703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:02.519884 32698 server_base.cc:1034] running on GCE node
I20240501 02:09:02.520061 32698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:02.520255 32698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:02.521392 32698 hybrid_clock.cc:648] HybridClock initialized: now 1714529342521381 us; error 27 us; skew 500 ppm
I20240501 02:09:02.522725 32698 webserver.cc:460] Webserver started at http://127.22.12.67:41379/ using document root <none> and password file <none>
I20240501 02:09:02.522904 32698 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:02.522943 32698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:02.523044 32698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:02.524031 32698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "edfdb4b3a0de4546926f2e5a84e713a9"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "bc9fc13bffedc6e626ecf708aab196bb"
server_key_iv: "3101db3ac1874e1792c010a93f92a5b1"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.524307 32698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "edfdb4b3a0de4546926f2e5a84e713a9"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "bc9fc13bffedc6e626ecf708aab196bb"
server_key_iv: "3101db3ac1874e1792c010a93f92a5b1"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.525384 32698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:02.526010 32712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.526160 32698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:09:02.526227 32698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "edfdb4b3a0de4546926f2e5a84e713a9"
format_stamp: "Formatted at 2024-05-01 02:09:02 on dist-test-slave-9tkh"
server_key: "bc9fc13bffedc6e626ecf708aab196bb"
server_key_iv: "3101db3ac1874e1792c010a93f92a5b1"
server_key_version: "encryptionkey@0"
I20240501 02:09:02.526288 32698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:02.529744 32698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:02.529938 32698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:02.530016 32698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:02.530164 32698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:02.530400 32698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:02.530427 32698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.530452 32698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:02.530468 32698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:02.535647 32698 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:41847
I20240501 02:09:02.535686 357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:41847 every 8 connection(s)
I20240501 02:09:02.535972 32698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:09:02.538435 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.536669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:39595 (local address 127.22.12.126:35787)
0501 02:09:02.536727 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.536729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.536924 (+ 195us) server_negotiation.cc:407] Connection header received
0501 02:09:02.537442 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.537443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.537449 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.537506 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.538192 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.538194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:02.538194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:09:02.538198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:02.538214 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:09:02.538218 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.538266 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:02.538339 (+ 73us) server_negotiation.cc:299] Negotiation successful
0501 02:09:02.538351 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 02:09:02.538614 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:02.536651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35787 (local address 127.22.12.67:39595)
0501 02:09:02.536815 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.536838 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:02.537334 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:02.537566 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:02.537570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:02.537578 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:02.538070 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:09:02.538071 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:02.538089 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:09:02.538111 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:02.538269 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:02.538273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:02.538276 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:09:02.538370 (+ 94us) client_negotiation.cc:240] Negotiation successful
0501 02:09:02.538422 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20240501 02:09:02.539227 358 heartbeater.cc:347] Connected to a master server at 127.22.12.126:35787
I20240501 02:09:02.539319 358 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:02.539445 358 heartbeater.cc:511] Master 127.22.12.126:35787 requested a full tablet report, sending...
I20240501 02:09:02.539821 32376 ts_manager.cc:194] Registered new tserver with Master: edfdb4b3a0de4546926f2e5a84e713a9 (127.22.12.67:41847)
I20240501 02:09:02.543215 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 32698
I20240501 02:09:02.543308 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:09:02.543457 22577 external_mini_cluster.cc:1428] Setting key 96b5eb11d5c7eccc0cc6dd22809bbc91
I20240501 02:09:02.544236 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240501 02:09:02.545855 32416 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:09:02.545416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.22.12.126:35787)
0501 02:09:02.545478 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:02.545479 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:02.545571 (+ 92us) server_negotiation.cc:407] Connection header received
0501 02:09:02.545632 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:02.545633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:02.545638 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.545685 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:02.545794 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39548: BlockingRecv error: recv got EOF from 127.0.0.1:39548 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 02:09:02.545922 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 32436
I20240501 02:09:02.550937 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 32567
I20240501 02:09:02.554837 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 32698
I20240501 02:09:02.559401 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 32345
2024-05-01T02:09:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:02 dist-test-slave-9tkh krb5kdc[368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:02 dist-test-slave-9tkh krb5kdc[368](info): set up 2 sockets
May 01 02:09:02 dist-test-slave-9tkh krb5kdc[368](info): commencing operation
krb5kdc: starting...
W20240501 02:09:03.450649 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:09:03 dist-test-slave-9tkh krb5kdc[368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:09:03.562363 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:42687
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:03.642585 381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:03.642789 381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:03.642810 381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:03.644151 381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:09:03.644196 381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:03.644209 381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:03.644219 381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:09:03.644230 381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:09:03.646169 381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:03.646428 381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:03.646659 381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:03.651252 386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:03.651800 389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:03.652137 387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:03.652619 381 server_base.cc:1034] running on GCE node
I20240501 02:09:03.652813 381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:03.653031 381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:03.654170 381 hybrid_clock.cc:648] HybridClock initialized: now 1714529343654155 us; error 30 us; skew 500 ppm
May 01 02:09:03 dist-test-slave-9tkh krb5kdc[368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:03.656644 381 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:09:03.657056 381 webserver.cc:460] Webserver started at http://127.22.12.126:39489/ using document root <none> and password file <none>
I20240501 02:09:03.657219 381 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:03.657253 381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:03.657333 381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:03.658258 381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "d118719b38184f49941294231f6ce48c"
format_stamp: "Formatted at 2024-05-01 02:09:03 on dist-test-slave-9tkh"
server_key: "7bf34ba1df9df4b829f038e2793c09a7"
server_key_iv: "e580342d7af376d23f708abd43277e21"
server_key_version: "encryptionkey@0"
I20240501 02:09:03.658501 381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "d118719b38184f49941294231f6ce48c"
format_stamp: "Formatted at 2024-05-01 02:09:03 on dist-test-slave-9tkh"
server_key: "7bf34ba1df9df4b829f038e2793c09a7"
server_key_iv: "e580342d7af376d23f708abd43277e21"
server_key_version: "encryptionkey@0"
I20240501 02:09:03.659665 381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:09:03.660387 396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:03.660573 381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:09:03.660646 381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "d118719b38184f49941294231f6ce48c"
format_stamp: "Formatted at 2024-05-01 02:09:03 on dist-test-slave-9tkh"
server_key: "7bf34ba1df9df4b829f038e2793c09a7"
server_key_iv: "e580342d7af376d23f708abd43277e21"
server_key_version: "encryptionkey@0"
I20240501 02:09:03.660727 381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:03.677759 381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:03.679996 381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:03.680109 381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:03.683794 381 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:42687
I20240501 02:09:03.683825 448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:42687 every 8 connection(s)
I20240501 02:09:03.684170 381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:09:03.684806 449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:03.686970 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 381
I20240501 02:09:03.687062 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:09:03.687206 22577 external_mini_cluster.cc:1428] Setting key 51d9618bf5b7de9203da12c85316238d
I20240501 02:09:03.687459 449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c: Bootstrap starting.
I20240501 02:09:03.688176 449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:03.688485 449 log.cc:824] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:03.689237 449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c: No bootstrap required, opened a new log
I20240501 02:09:03.691246 449 raft_consensus.cc:348] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d118719b38184f49941294231f6ce48c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 42687 } }
I20240501 02:09:03.691395 449 raft_consensus.cc:374] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:03.691422 449 raft_consensus.cc:729] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d118719b38184f49941294231f6ce48c, State: Initialized, Role: FOLLOWER
I20240501 02:09:03.691870 449 consensus_queue.cc:260] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d118719b38184f49941294231f6ce48c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 42687 } }
I20240501 02:09:03.691954 449 raft_consensus.cc:388] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:09:03.691977 449 raft_consensus.cc:482] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:09:03.692006 449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:03.692591 449 raft_consensus.cc:504] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d118719b38184f49941294231f6ce48c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 42687 } }
I20240501 02:09:03.692692 449 leader_election.cc:302] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d118719b38184f49941294231f6ce48c; no voters:
I20240501 02:09:03.692857 449 leader_election.cc:288] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:09:03.692972 454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:09:03.693121 449 sys_catalog.cc:564] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:09:03.693140 454 raft_consensus.cc:686] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [term 1 LEADER]: Becoming Leader. State: Replica: d118719b38184f49941294231f6ce48c, State: Running, Role: LEADER
May 01 02:09:03 dist-test-slave-9tkh krb5kdc[368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:03.693334 454 consensus_queue.cc:237] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d118719b38184f49941294231f6ce48c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 42687 } }
I20240501 02:09:03.693642 454 sys_catalog.cc:455] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d118719b38184f49941294231f6ce48c. Latest consensus state: current_term: 1 leader_uuid: "d118719b38184f49941294231f6ce48c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d118719b38184f49941294231f6ce48c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 42687 } } }
I20240501 02:09:03.693714 454 sys_catalog.cc:458] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:03.693894 462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:09:03.693881 456 sys_catalog.cc:455] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d118719b38184f49941294231f6ce48c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d118719b38184f49941294231f6ce48c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 42687 } } }
I20240501 02:09:03.693990 456 sys_catalog.cc:458] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:03.694550 462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:09:03.697239 462 catalog_manager.cc:1347] Generated new cluster ID: f109d45220bf410fbbbcf44be1693480
I20240501 02:09:03.697320 462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:09:03.697376 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:03.687743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.22.12.126:42687)
0501 02:09:03.687956 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:03.687968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:03.687993 (+ 25us) server_negotiation.cc:407] Connection header received
0501 02:09:03.688588 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:03.688597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:03.688661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.688810 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:03.689574 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.690770 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.691362 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.691607 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.693919 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.693930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:03.693930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:03.693953 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:03.695795 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.696192 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.696195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:03.696195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:03.696261 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.696438 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.696440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:03.696440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:03.696695 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.696822 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:03.697027 (+ 205us) server_negotiation.cc:299] Negotiation successful
0501 02:09:03.697130 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20240501 02:09:03.720341 462 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:09:03.720924 462 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:09:03.729105 462 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c: Generated new TSK 0
I20240501 02:09:03.729347 462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:09:03.761353 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:42687
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:03.840237 475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:03.840440 475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:03.840462 475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:03.841802 475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:03.841861 475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:03.841887 475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:09:03.843427 475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:03.843647 475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:03.843863 475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:03.848604 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:03.849002 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:03.849362 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:03.849784 475 server_base.cc:1034] running on GCE node
I20240501 02:09:03.849956 475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:03.850159 475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:03.851294 475 hybrid_clock.cc:648] HybridClock initialized: now 1714529343851281 us; error 32 us; skew 500 ppm
May 01 02:09:03 dist-test-slave-9tkh krb5kdc[368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:03.853545 475 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:09:03.853964 475 webserver.cc:460] Webserver started at http://127.22.12.65:43053/ using document root <none> and password file <none>
I20240501 02:09:03.854130 475 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:03.854166 475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:03.854265 475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:03.855254 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "9ad5dfdf1d5f4bb291c7916dabae8589"
format_stamp: "Formatted at 2024-05-01 02:09:03 on dist-test-slave-9tkh"
server_key: "0bd953ab817ba73302c82c8e53dfedb5"
server_key_iv: "a56d090cecbfd2a2f0884447533399d6"
server_key_version: "encryptionkey@0"
I20240501 02:09:03.855535 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "9ad5dfdf1d5f4bb291c7916dabae8589"
format_stamp: "Formatted at 2024-05-01 02:09:03 on dist-test-slave-9tkh"
server_key: "0bd953ab817ba73302c82c8e53dfedb5"
server_key_iv: "a56d090cecbfd2a2f0884447533399d6"
server_key_version: "encryptionkey@0"
I20240501 02:09:03.856607 475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:03.857242 490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:03.857424 475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:09:03.857497 475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "9ad5dfdf1d5f4bb291c7916dabae8589"
format_stamp: "Formatted at 2024-05-01 02:09:03 on dist-test-slave-9tkh"
server_key: "0bd953ab817ba73302c82c8e53dfedb5"
server_key_iv: "a56d090cecbfd2a2f0884447533399d6"
server_key_version: "encryptionkey@0"
I20240501 02:09:03.857547 475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:03.870988 475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:03.872987 475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:03.873090 475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:03.873275 475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:03.873538 475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:03.873565 475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:03.873585 475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:03.873595 475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:03.878844 475 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:43517
I20240501 02:09:03.878914 603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:43517 every 8 connection(s)
I20240501 02:09:03.879186 475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:09:03 dist-test-slave-9tkh krb5kdc[368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:03.885967 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 475
I20240501 02:09:03.886086 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:09:03.886282 22577 external_mini_cluster.cc:1428] Setting key 21f37981ab518d1928e206a479f5c79f
I20240501 02:09:03.887882 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:03.879930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:59329 (local address 127.22.12.126:42687)
0501 02:09:03.880002 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:03.880004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:03.880183 (+ 179us) server_negotiation.cc:407] Connection header received
0501 02:09:03.880635 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:03.880636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:03.880646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.880731 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:03.881762 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.882258 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.882922 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.883047 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.884934 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.884939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:03.884939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:03.884946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:03.886393 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.886895 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.886897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:03.886897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:03.886926 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.887147 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.887148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:03.887148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:03.887282 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.887356 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:03.887651 (+ 295us) server_negotiation.cc:299] Negotiation successful
0501 02:09:03.887769 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 02:09:03.887907 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:03.879863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:42687 (local address 127.22.12.65:59329)
0501 02:09:03.880042 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.880070 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:03.880529 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.880793 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:03.880797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:03.881095 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:03.881647 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:03.881652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.882346 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.882347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:03.882837 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:03.882839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.882953 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.883456 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:03.883472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:03.884848 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:03.886472 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:03.886480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:03.886483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:03.886805 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:03.886996 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:03.886996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:03.886997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:03.887083 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:03.887372 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:03.887378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:03.887514 (+ 136us) client_negotiation.cc:769] Sending connection context
0501 02:09:03.887617 (+ 103us) client_negotiation.cc:240] Negotiation successful
0501 02:09:03.887734 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20240501 02:09:03.888971 604 heartbeater.cc:347] Connected to a master server at 127.22.12.126:42687
I20240501 02:09:03.889081 604 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:03.889300 604 heartbeater.cc:511] Master 127.22.12.126:42687 requested a full tablet report, sending...
I20240501 02:09:03.889816 413 ts_manager.cc:194] Registered new tserver with Master: 9ad5dfdf1d5f4bb291c7916dabae8589 (127.22.12.65:43517)
I20240501 02:09:03.890470 413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:59329
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:09:03.910719 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:42687
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:03.988584 609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:03.988787 609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:03.988817 609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:03.990123 609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:03.990164 609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:03.990187 609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:09:03.991752 609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:03.991946 609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:03.992100 609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:03.996801 617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:03.997012 614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:03.997390 615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:03.997817 609 server_base.cc:1034] running on GCE node
I20240501 02:09:03.997987 609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:03.998194 609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:03.999328 609 hybrid_clock.cc:648] HybridClock initialized: now 1714529343999318 us; error 27 us; skew 500 ppm
May 01 02:09:03 dist-test-slave-9tkh krb5kdc[368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:04.001704 609 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:09:04.002133 609 webserver.cc:460] Webserver started at http://127.22.12.66:44227/ using document root <none> and password file <none>
I20240501 02:09:04.002296 609 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:04.002336 609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:04.002431 609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:04.003451 609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "0b9d89b7c1a74306849b23407288a987"
format_stamp: "Formatted at 2024-05-01 02:09:04 on dist-test-slave-9tkh"
server_key: "1a65d78c38fd1683188e6c69b81b8f94"
server_key_iv: "1fdd4b4b17fc2d9b8e2182b7e6a076ec"
server_key_version: "encryptionkey@0"
I20240501 02:09:04.003772 609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "0b9d89b7c1a74306849b23407288a987"
format_stamp: "Formatted at 2024-05-01 02:09:04 on dist-test-slave-9tkh"
server_key: "1a65d78c38fd1683188e6c69b81b8f94"
server_key_iv: "1fdd4b4b17fc2d9b8e2182b7e6a076ec"
server_key_version: "encryptionkey@0"
I20240501 02:09:04.004887 609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:09:04.005487 624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:04.005633 609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:09:04.005705 609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "0b9d89b7c1a74306849b23407288a987"
format_stamp: "Formatted at 2024-05-01 02:09:04 on dist-test-slave-9tkh"
server_key: "1a65d78c38fd1683188e6c69b81b8f94"
server_key_iv: "1fdd4b4b17fc2d9b8e2182b7e6a076ec"
server_key_version: "encryptionkey@0"
I20240501 02:09:04.005760 609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:04.016652 609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:04.018625 609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:04.018733 609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:04.018929 609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:04.019213 609 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:04.019244 609 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:04.019273 609 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:04.019299 609 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:04.024196 609 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:42059
I20240501 02:09:04.024250 737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:42059 every 8 connection(s)
I20240501 02:09:04.024514 609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:09:04.025066 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 609
I20240501 02:09:04.025161 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:09:04.025336 22577 external_mini_cluster.cc:1428] Setting key 304ffda612d73ca932a446439231a5be
May 01 02:09:04 dist-test-slave-9tkh krb5kdc[368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529343, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
I20240501 02:09:04.032397 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.025117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:51749 (local address 127.22.12.126:42687)
0501 02:09:04.025166 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.025168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.025347 (+ 179us) server_negotiation.cc:407] Connection header received
0501 02:09:04.025832 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.025833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.025844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.025908 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:04.026889 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.027372 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.027889 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.028044 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.029933 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:04.029937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:04.029937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:04.029942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:04.031273 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:04.031608 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:04.031609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:04.031610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:04.031632 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:04.031788 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:04.031789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:04.031789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:04.031881 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:04.031935 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.032191 (+ 256us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.032285 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 02:09:04.032474 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.025088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:42687 (local address 127.22.12.66:51749)
0501 02:09:04.025238 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.025267 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.025739 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.025970 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.025976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.026266 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:04.026803 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.026809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.027457 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.027458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.027819 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.027821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.027950 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.028473 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:04.028488 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:04.029832 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:04.031347 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:04.031353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:04.031356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:04.031546 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:04.031683 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:04.031684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:04.031684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:04.031738 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:04.031931 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:04.031936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:04.032070 (+ 134us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.032164 (+ 94us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.032289 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20240501 02:09:04.032913 738 heartbeater.cc:347] Connected to a master server at 127.22.12.126:42687
I20240501 02:09:04.033012 738 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:04.033214 738 heartbeater.cc:511] Master 127.22.12.126:42687 requested a full tablet report, sending...
I20240501 02:09:04.033612 413 ts_manager.cc:194] Registered new tserver with Master: 0b9d89b7c1a74306849b23407288a987 (127.22.12.66:42059)
I20240501 02:09:04.034045 413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:51749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:09:04.049180 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:42687
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:04.127316 743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:04.127521 743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:04.127543 743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:04.128890 743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:04.128947 743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:04.128973 743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:09:04.130472 743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:04.130712 743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:04.130905 743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:04.135293 751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:04.135571 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:04.135963 748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:04.136402 743 server_base.cc:1034] running on GCE node
I20240501 02:09:04.136565 743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:04.136749 743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:04.137883 743 hybrid_clock.cc:648] HybridClock initialized: now 1714529344137869 us; error 32 us; skew 500 ppm
May 01 02:09:04 dist-test-slave-9tkh krb5kdc[368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529344, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:04.140231 743 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:09:04.140635 743 webserver.cc:460] Webserver started at http://127.22.12.67:44727/ using document root <none> and password file <none>
I20240501 02:09:04.140803 743 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:04.140841 743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:04.140955 743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:09:04.141894 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "e117d86b66134a32bf23f8f56a27c143"
format_stamp: "Formatted at 2024-05-01 02:09:04 on dist-test-slave-9tkh"
server_key: "a9561257e654ae0b9f697e2f6fee0313"
server_key_iv: "9b7c327a46eed2b6c8c6b4a5523641aa"
server_key_version: "encryptionkey@0"
I20240501 02:09:04.142175 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "e117d86b66134a32bf23f8f56a27c143"
format_stamp: "Formatted at 2024-05-01 02:09:04 on dist-test-slave-9tkh"
server_key: "a9561257e654ae0b9f697e2f6fee0313"
server_key_iv: "9b7c327a46eed2b6c8c6b4a5523641aa"
server_key_version: "encryptionkey@0"
I20240501 02:09:04.143309 743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:09:04.143930 758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:04.144073 743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:09:04.144143 743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "e117d86b66134a32bf23f8f56a27c143"
format_stamp: "Formatted at 2024-05-01 02:09:04 on dist-test-slave-9tkh"
server_key: "a9561257e654ae0b9f697e2f6fee0313"
server_key_iv: "9b7c327a46eed2b6c8c6b4a5523641aa"
server_key_version: "encryptionkey@0"
I20240501 02:09:04.144204 743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:04.177642 743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:04.179796 743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:04.179904 743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:04.180124 743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:04.180411 743 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:04.180444 743 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:04.180480 743 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:04.180502 743 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:04.185869 743 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:38445
I20240501 02:09:04.185915 871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:38445 every 8 connection(s)
I20240501 02:09:04.186228 743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:09:04 dist-test-slave-9tkh krb5kdc[368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529344, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:04.193732 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 743
I20240501 02:09:04.193863 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:09:04.194079 22577 external_mini_cluster.cc:1428] Setting key 837c387dcc7e8421b543540545c42939
I20240501 02:09:04.194968 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.186970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:35921 (local address 127.22.12.126:42687)
0501 02:09:04.187066 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.187068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.187222 (+ 154us) server_negotiation.cc:407] Connection header received
0501 02:09:04.187712 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.187713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.187752 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.187804 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:04.188833 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.189339 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.189991 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.190167 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.192229 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:04.192233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:04.192234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:04.192241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:04.193465 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:04.193975 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:04.193976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:04.193976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:04.194016 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:04.194221 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:04.194222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:04.194222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:04.194351 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:04.194418 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.194775 (+ 357us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.194877 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20240501 02:09:04.195077 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.186931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:42687 (local address 127.22.12.67:35921)
0501 02:09:04.187113 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.187140 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.187608 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.187872 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.187876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.188188 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:04.188731 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.188736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.189422 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.189423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.189889 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.189893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.190042 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.190693 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:04.190711 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:04.192121 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:04.193553 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:04.193559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:04.193563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:04.193871 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:04.194081 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:04.194081 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:04.194082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:04.194168 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:04.194407 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:04.194413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:04.194597 (+ 184us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.194721 (+ 124us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.194874 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20240501 02:09:04.195544 872 heartbeater.cc:347] Connected to a master server at 127.22.12.126:42687
I20240501 02:09:04.195644 872 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:04.195852 872 heartbeater.cc:511] Master 127.22.12.126:42687 requested a full tablet report, sending...
I20240501 02:09:04.196194 413 ts_manager.cc:194] Registered new tserver with Master: e117d86b66134a32bf23f8f56a27c143 (127.22.12.67:38445)
I20240501 02:09:04.196557 413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:35921
I20240501 02:09:04.205812 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:09:04.211887 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.207049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.22.12.126:42687)
0501 02:09:04.207107 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.207109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.207180 (+ 71us) server_negotiation.cc:407] Connection header received
0501 02:09:04.207220 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.207221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.207230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.207287 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:04.207877 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.208351 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.208884 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.209004 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.209891 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:04.209895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:04.209895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:04.209901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:04.211168 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:04.211390 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:04.211391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:04.211391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:04.211408 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:04.211495 (+ 87us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:04.211495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:04.211495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:04.211601 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:04.211671 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.211735 (+ 64us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.211816 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 02:09:04.213383 413 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:09:04.214812 413 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:09:04.219763 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.216668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:43517 (local address 127.0.0.1:35542)
0501 02:09:04.216783 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.216809 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.216893 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.217256 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.217259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.217263 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:04.217522 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.217524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.218441 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.218442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.219210 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.219212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.219286 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.219325 (+ 39us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.219649 (+ 324us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.219689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20240501 02:09:04.219925 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.216668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:38445 (local address 127.0.0.1:50800)
0501 02:09:04.217099 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.217105 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.217144 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.217372 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.217373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.217375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:04.217590 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.217591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.218908 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.218909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.219740 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.219743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.219799 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.219803 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.219829 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.219862 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20240501 02:09:04.220155 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.216896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.22.12.65:43517)
0501 02:09:04.217083 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.217091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.217101 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:09:04.217140 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.217142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.217160 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.217237 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:04.217608 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.218363 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.219335 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.219938 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.219994 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.220027 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.220073 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20240501 02:09:04.220242 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.216671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:42059 (local address 127.0.0.1:47076)
0501 02:09:04.216987 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.216994 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.217040 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.217349 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.217350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.217352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:04.217584 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.217585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.219016 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.219017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.219815 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.219818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.220107 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.220111 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.220139 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.220181 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":63,"threads_started":1}
I20240501 02:09:04.220758 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.216835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.22.12.67:38445)
0501 02:09:04.217019 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.217027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.217190 (+ 163us) server_negotiation.cc:407] Connection header received
0501 02:09:04.217228 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.217230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.217245 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.217306 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:04.217661 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.218829 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.220077 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.220615 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.220649 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.220677 (+ 28us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.220705 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20240501 02:09:04.220742 538 tablet_service.cc:1467] Processing CreateTablet for tablet bd81536800514211939e6c120b4fd842 (DEFAULT_TABLE table=test-table [id=b363a503e6a143c5afea7222c43c5620]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:09:04.221038 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd81536800514211939e6c120b4fd842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:04.221112 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.217054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.22.12.66:42059)
0501 02:09:04.217189 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.217194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.217202 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:09:04.217239 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.217240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.217257 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.217334 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:04.217679 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.218844 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.220416 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.220928 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.220968 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.220999 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.221039 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20240501 02:09:04.221230 806 tablet_service.cc:1467] Processing CreateTablet for tablet bd81536800514211939e6c120b4fd842 (DEFAULT_TABLE table=test-table [id=b363a503e6a143c5afea7222c43c5620]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:09:04.221473 806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd81536800514211939e6c120b4fd842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:04.223060 672 tablet_service.cc:1467] Processing CreateTablet for tablet bd81536800514211939e6c120b4fd842 (DEFAULT_TABLE table=test-table [id=b363a503e6a143c5afea7222c43c5620]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:09:04.223335 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd81536800514211939e6c120b4fd842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:04.223584 886 tablet_bootstrap.cc:492] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: Bootstrap starting.
I20240501 02:09:04.224123 886 tablet_bootstrap.cc:654] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:04.224201 887 tablet_bootstrap.cc:492] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Bootstrap starting.
I20240501 02:09:04.224534 886 log.cc:824] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:04.224756 887 tablet_bootstrap.cc:654] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:04.224995 887 log.cc:824] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:04.225344 886 tablet_bootstrap.cc:492] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: No bootstrap required, opened a new log
I20240501 02:09:04.225400 886 ts_tablet_manager.cc:1367] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:09:04.225697 887 tablet_bootstrap.cc:492] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: No bootstrap required, opened a new log
I20240501 02:09:04.225767 887 ts_tablet_manager.cc:1367] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:09:04.225976 890 tablet_bootstrap.cc:492] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987: Bootstrap starting.
I20240501 02:09:04.226558 890 tablet_bootstrap.cc:654] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:04.226805 890 log.cc:824] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:04.227325 890 tablet_bootstrap.cc:492] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987: No bootstrap required, opened a new log
I20240501 02:09:04.227376 890 ts_tablet_manager.cc:1367] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:09:04.227527 886 raft_consensus.cc:348] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.227669 886 raft_consensus.cc:374] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:04.227692 886 raft_consensus.cc:729] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad5dfdf1d5f4bb291c7916dabae8589, State: Initialized, Role: FOLLOWER
I20240501 02:09:04.227813 886 consensus_queue.cc:260] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.227849 887 raft_consensus.cc:348] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.227963 887 raft_consensus.cc:374] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:04.227993 887 raft_consensus.cc:729] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e117d86b66134a32bf23f8f56a27c143, State: Initialized, Role: FOLLOWER
I20240501 02:09:04.228060 886 ts_tablet_manager.cc:1398] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:09:04.228085 887 consensus_queue.cc:260] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.228170 604 heartbeater.cc:503] Master 127.22.12.126:42687 was elected leader, sending a full tablet report...
I20240501 02:09:04.228302 887 ts_tablet_manager.cc:1398] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 02:09:04.228319 872 heartbeater.cc:503] Master 127.22.12.126:42687 was elected leader, sending a full tablet report...
I20240501 02:09:04.229154 890 raft_consensus.cc:348] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.229275 890 raft_consensus.cc:374] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:04.229300 890 raft_consensus.cc:729] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9d89b7c1a74306849b23407288a987, State: Initialized, Role: FOLLOWER
I20240501 02:09:04.229393 890 consensus_queue.cc:260] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.229621 890 ts_tablet_manager.cc:1398] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:09:04.229734 738 heartbeater.cc:503] Master 127.22.12.126:42687 was elected leader, sending a full tablet report...
I20240501 02:09:04.236790 893 raft_consensus.cc:482] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:09:04.236928 893 raft_consensus.cc:504] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.237250 893 leader_election.cc:288] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9d89b7c1a74306849b23407288a987 (127.22.12.66:42059), 9ad5dfdf1d5f4bb291c7916dabae8589 (127.22.12.65:43517)
I20240501 02:09:04.239485 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.237424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:43517 (local address 127.22.12.67:43493)
0501 02:09:04.237482 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.237493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.237536 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.237680 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.237681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.237683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:04.237923 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.237924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.238606 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.238608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.239286 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.239288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.239349 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.239354 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.239378 (+ 24us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.239414 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240501 02:09:04.239929 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.237497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:43493 (local address 127.22.12.65:43517)
0501 02:09:04.237557 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.237559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.237568 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:09:04.237591 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.237592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.237602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.237677 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:04.238001 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.238550 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.239459 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.239824 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.239831 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.239849 (+ 18us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.239870 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 02:09:04.240131 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.237483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:47663 (local address 127.22.12.66:42059)
0501 02:09:04.237527 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.237529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.237729 (+ 200us) server_negotiation.cc:407] Connection header received
0501 02:09:04.237774 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.237775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.237785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.237831 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:04.238170 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.238718 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.239519 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.239851 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.239857 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.240027 (+ 170us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.240066 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:09:04.240267 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd81536800514211939e6c120b4fd842" candidate_uuid: "e117d86b66134a32bf23f8f56a27c143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" is_pre_election: true
I20240501 02:09:04.240330 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.237480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:42059 (local address 127.22.12.67:47663)
0501 02:09:04.237651 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.237661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:04.237720 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.237858 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:04.237859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:04.237861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:04.238081 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.238083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.238784 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.238785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:04.239425 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:04.239427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.239945 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.239948 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:09:04.240216 (+ 268us) client_negotiation.cc:240] Negotiation successful
0501 02:09:04.240258 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20240501 02:09:04.240402 558 raft_consensus.cc:2454] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e117d86b66134a32bf23f8f56a27c143 in term 0.
I20240501 02:09:04.240618 761 leader_election.cc:302] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad5dfdf1d5f4bb291c7916dabae8589, e117d86b66134a32bf23f8f56a27c143; no voters:
I20240501 02:09:04.240734 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd81536800514211939e6c120b4fd842" candidate_uuid: "e117d86b66134a32bf23f8f56a27c143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9d89b7c1a74306849b23407288a987" is_pre_election: true
I20240501 02:09:04.240787 893 raft_consensus.cc:2781] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:09:04.240850 893 raft_consensus.cc:482] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:09:04.240880 893 raft_consensus.cc:3037] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:04.240895 692 raft_consensus.cc:2454] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e117d86b66134a32bf23f8f56a27c143 in term 0.
I20240501 02:09:04.241540 893 raft_consensus.cc:504] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.241664 893 leader_election.cc:288] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [CANDIDATE]: Term 1 election: Requested vote from peers 0b9d89b7c1a74306849b23407288a987 (127.22.12.66:42059), 9ad5dfdf1d5f4bb291c7916dabae8589 (127.22.12.65:43517)
I20240501 02:09:04.241796 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd81536800514211939e6c120b4fd842" candidate_uuid: "e117d86b66134a32bf23f8f56a27c143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9d89b7c1a74306849b23407288a987"
I20240501 02:09:04.241850 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd81536800514211939e6c120b4fd842" candidate_uuid: "e117d86b66134a32bf23f8f56a27c143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589"
I20240501 02:09:04.241863 692 raft_consensus.cc:3037] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:04.241937 558 raft_consensus.cc:3037] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:04.242566 692 raft_consensus.cc:2454] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e117d86b66134a32bf23f8f56a27c143 in term 1.
I20240501 02:09:04.242636 558 raft_consensus.cc:2454] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e117d86b66134a32bf23f8f56a27c143 in term 1.
I20240501 02:09:04.242743 762 leader_election.cc:302] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9d89b7c1a74306849b23407288a987, e117d86b66134a32bf23f8f56a27c143; no voters:
I20240501 02:09:04.242841 893 raft_consensus.cc:2781] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:09:04.242983 893 raft_consensus.cc:686] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 1 LEADER]: Becoming Leader. State: Replica: e117d86b66134a32bf23f8f56a27c143, State: Running, Role: LEADER
I20240501 02:09:04.243072 893 consensus_queue.cc:237] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } }
I20240501 02:09:04.243868 412 catalog_manager.cc:5484] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 reported cstate change: term changed from 0 to 1, leader changed from <none> to e117d86b66134a32bf23f8f56a27c143 (127.22.12.67). New cstate: current_term: 1 leader_uuid: "e117d86b66134a32bf23f8f56a27c143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117d86b66134a32bf23f8f56a27c143" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 38445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 } health_report { overall_health: UNKNOWN } } }
I20240501 02:09:04.249478 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.247542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.22.12.67:38445)
0501 02:09:04.247589 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:04.247591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:04.247599 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:09:04.247653 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:04.247653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:04.247669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.247733 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:09:04.247972 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.248454 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.249016 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.249152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.249186 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:04.249265 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:04.249331 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:04.249388 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 02:09:04.249409 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:09:04.251065 558 raft_consensus.cc:1261] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589 [term 1 FOLLOWER]: Refusing update from remote peer e117d86b66134a32bf23f8f56a27c143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:09:04.251065 692 raft_consensus.cc:1261] T bd81536800514211939e6c120b4fd842 P 0b9d89b7c1a74306849b23407288a987 [term 1 FOLLOWER]: Refusing update from remote peer e117d86b66134a32bf23f8f56a27c143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:09:04.251363 893 consensus_queue.cc:1035] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad5dfdf1d5f4bb291c7916dabae8589" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:09:04.251469 896 consensus_queue.cc:1035] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9d89b7c1a74306849b23407288a987" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:09:04.253926 901 mvcc.cc:205] Tried to move back new op lower bound from 7022712194049658880 to 7022712194020265984. Current Snapshot: MvccSnapshot[applied={T|T < 7022712194049658880}]
I20240501 02:09:04.254065 900 mvcc.cc:205] Tried to move back new op lower bound from 7022712194049658880 to 7022712194020265984. Current Snapshot: MvccSnapshot[applied={T|T < 7022712194049658880}]
I20240501 02:09:04.254667 899 mvcc.cc:205] Tried to move back new op lower bound from 7022712194049658880 to 7022712194020265984. Current Snapshot: MvccSnapshot[applied={T|T < 7022712194049658880}]
I20240501 02:09:04.256532 412 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52996:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:09:04.256616 412 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52996:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:09:04.257259 412 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d118719b38184f49941294231f6ce48c: Sending DeleteTablet for 3 replicas of tablet bd81536800514211939e6c120b4fd842
I20240501 02:09:04.257567 672 tablet_service.cc:1514] Processing DeleteTablet for tablet bd81536800514211939e6c120b4fd842 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:09:04 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:47076
I20240501 02:09:04.257620 538 tablet_service.cc:1514] Processing DeleteTablet for tablet bd81536800514211939e6c120b4fd842 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:09:04 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:35542
I20240501 02:09:04.257663 806 tablet_service.cc:1514] Processing DeleteTablet for tablet bd81536800514211939e6c120b4fd842 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:09:04 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50800
I20240501 02:09:04.257836 911 tablet_replica.cc:330] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: stopping tablet replica
I20240501 02:09:04.257912 911 raft_consensus.cc:2229] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:09:04.258019 911 raft_consensus.cc:2258] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:09:04.258087 910 tablet_replica.cc:330] T bd81536800514211939e6c120b4fd842 P 9ad5dfdf1d5f4bb291c7916dabae8589: stopping tablet replica
I20240501 02:09:04.258167 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 475
I20240501 02:09:04.259171 911 ts_tablet_manager.cc:1875] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:09:04.260802 911 ts_tablet_manager.cc:1888] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:09:04.260869 911 log.cc:1196] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/bd81536800514211939e6c120b4fd842
I20240501 02:09:04.261122 911 ts_tablet_manager.cc:1909] T bd81536800514211939e6c120b4fd842 P e117d86b66134a32bf23f8f56a27c143: Deleting consensus metadata
I20240501 02:09:04.261479 398 catalog_manager.cc:4830] TS e117d86b66134a32bf23f8f56a27c143 (127.22.12.67:38445): tablet bd81536800514211939e6c120b4fd842 (table test-table [id=b363a503e6a143c5afea7222c43c5620]) successfully deleted
W20240501 02:09:04.263823 399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:43517 (error 108)
I20240501 02:09:04.264436 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 609
I20240501 02:09:04.265178 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.264319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:43517 (local address 127.0.0.1:35552)
0501 02:09:04.265104 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.265145 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:43517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":764}
W20240501 02:09:04.266098 399 catalog_manager.cc:4573] TS 9ad5dfdf1d5f4bb291c7916dabae8589 (127.22.12.65:43517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd81536800514211939e6c120b4fd842: Network error: Client connection negotiation failed: client connection to 127.22.12.65:43517: connect: Connection refused (error 111)
I20240501 02:09:04.269455 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 743
I20240501 02:09:04.269469 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:04.269344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:42059 (local address 127.0.0.1:47084)
0501 02:09:04.269399 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.269429 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.66:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20240501 02:09:04.273707 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 381
2024-05-01T02:09:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1715 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:04 dist-test-slave-9tkh krb5kdc[915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:04 dist-test-slave-9tkh krb5kdc[915](info): set up 2 sockets
May 01 02:09:04 dist-test-slave-9tkh krb5kdc[915](info): commencing operation
krb5kdc: starting...
W20240501 02:09:05.170601 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.877s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:09:05 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:09:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/rangeradmin_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/rangeradmin_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/rangerlookup_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/rangerlookup_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
I20240501 02:09:05.329759 22577 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBKBEtG/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/postgres -l logfile start
2024-05-01 02:09:07.756 UTC [945] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-01 02:09:07.756 UTC [945] LOG: listening on IPv4 address "127.22.12.84", port 45213
2024-05-01 02:09:07.759 UTC [945] LOG: listening on Unix socket "/tmp/.s.PGSQL.45213"
2024-05-01 02:09:07.788 UTC [948] LOG: database system was shut down at 2024-05-01 02:09:06 UTC
2024-05-01 02:09:07.791 UTC [945] LOG: database system is ready to accept connections
I20240501 02:09:08.598488 22577 mini_postgres.cc:96] Postgres bound to 45213
2024-05-01 02:09:08.603 UTC [956] FATAL: database "slave" does not exist
127.22.12.84:45213 - accepting connections
I20240501 02:09:08.604403 22577 mini_ranger.cc:161] Starting Ranger...
I20240501 02:09:08.618158 22577 mini_ranger.cc:84] Created miniranger Postgres user
I20240501 02:09:09.069906 22577 mini_ranger.cc:87] Created ranger Postgres database
I20240501 02:09:09.069985 22577 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-admin
2024-05-01 02:09:09,281 [I] DB FLAVOR :POSTGRES
2024-05-01 02:09:09,282 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 02:09:09,282 [I] Checking connection..
2024-05-01 02:09:09,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-01 02:09:09,517 [I] Checking connection passed.
2024-05-01 02:09:09,517 [I] --------- Verifying version history table ---------
2024-05-01 02:09:09,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 02:09:09,760 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 02:09:09,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 02:09:10,016 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 02:09:10,016 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-01 02:09:10,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-01 02:09:10.260 UTC [1020] WARNING: there is no transaction in progress
2024-05-01 02:09:10,266 [I] create_dbversion_catalog.sql file imported successfully
2024-05-01 02:09:10,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 02:09:10,520 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-01 02:09:10,520 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 02:09:10,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-01 02:09:10,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-01 02:09:11,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tkh.c.gcp-upstream.internal','N') ;"
2024-05-01 02:09:11,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 02:09:11,520 [I] Table x_portal_user does not exist in database ranger
2024-05-01 02:09:11,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 02:09:11,775 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-01 02:09:11,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 02:09:12,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 02:09:12,272 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-01 02:09:12,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-01 02:09:13.272 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.291 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.309 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.326 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.343 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.360 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.418 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.425 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.439 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.450 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.460 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.474 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.485 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.493 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.500 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:13.509 UTC [1128] WARNING: there is no transaction in progress
2024-05-01 02:09:14,173 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-01 02:09:14,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-01 02:09:14,423 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-01 02:09:14,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 02:09:14,678 [I] Table x_portal_user already exists in database 'ranger'
2024-05-01 02:09:14,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 02:09:14,910 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-01 02:09:14,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 02:09:15,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 02:09:15,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tkh.c.gcp-upstream.internal';"
2024-05-01 02:09:15,662 [I] CORE_DB_SCHEMA import status has been updated
2024-05-01 02:09:15,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-01 02:09:15,913 [I] DB_PATCHES have already been applied
I20240501 02:09:15.919194 22577 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBKBEtG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240501 02:09:15.919266 22577 mini_ranger.cc:193] Using host: 127.22.12.84
I20240501 02:09:15.921846 22577 mini_ranger.cc:227] Ranger admin URL: http://127.22.12.84:38579
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp] .....
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp].
May 01, 2024 2:09:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240501 02:09:17.812305 22577 mini_ranger.cc:160] Time spent starting Ranger: real 9.208s user 0.000s sys 0.010s
May 01, 2024 2:09:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 01, 2024 2:09:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.MetricsREST
May 01, 2024 2:09:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 01, 2024 2:09:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 01, 2024 2:09:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240501 02:09:37.978760 22577 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/rangerkms_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/rangerkms_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 01 02:09:38 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529378, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240501 02:09:38.053010 22577 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240501 02:09:38.061530 22577 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240501 02:09:38.880693 22577 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240501 02:09:38.880770 22577 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms
I20240501 02:09:38.880787 22577 mini_ranger_kms.cc:227] Using postgres at 127.22.12.84:45213
2024-05-01 02:09:39,036 [I] DB FLAVOR :POSTGRES
2024-05-01 02:09:39,036 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 02:09:39,036 [I] Checking connection
2024-05-01 02:09:39,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-01 02:09:39,282 [I] connection success
2024-05-01 02:09:39,282 [I] --------- Verifying Ranger DB tables ---------
2024-05-01 02:09:39,282 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-01 02:09:39,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-01 02:09:39,540 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-01 02:09:39,540 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 02:09:39,540 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-01 02:09:39,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:45213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-01 02:09:39,817 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240501 02:09:40.332700 22577 mini_ranger_kms.cc:310] Created kms service
I20240501 02:09:40.512609 22577 mini_ranger_kms.cc:326] Created kudu user
I20240501 02:09:40.558833 22577 mini_ranger_kms.cc:343] Created rangerkms user
May 01 02:09:40 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529378, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:09:40.950594 22577 mini_ranger_kms.cc:384] Added ranger policy
I20240501 02:09:40.950712 22577 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240501 02:09:40.950731 22577 mini_ranger_kms.cc:240] Using host: 127.22.12.84
I20240501 02:09:40.953693 22577 mini_ranger_kms.cc:276] Ranger KMS PID: 1309
I20240501 02:09:40.953806 22577 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.12.84:36069
May 01, 2024 2:09:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 2:09:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 2:09:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 01, 2024 2:09:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/ews/webapp] .....
May 01, 2024 2:09:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-kms/ews/webapp].
May 01, 2024 2:09:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240501 02:09:42.810454 22577 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.757s user 0.009s sys 0.131s
I20240501 02:09:42.810588 22577 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240501 02:09:42.810649 22577 mini_ranger_kms.cc:401] 127.22.12.84:36069/kms/v1/keys
May 01 02:09:43 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714529383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 01 02:09:43 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.12.84@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
May 01 02:09:44 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.12.84@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
May 01 02:09:46 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:09:46.221741 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44031
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:44031
--ranger_config_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:36069/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:46.307111 1360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:46.307322 1360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:46.307343 1360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:46.308816 1360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:09:46.308872 1360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:46.308884 1360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:46.308895 1360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:09:46.308905 1360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:09:46.310750 1360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.12.84:36069/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:44031
--ranger_config_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:46.310968 1360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:46.311165 1360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:46.316336 1366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:46.316673 1365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:46.317183 1360 server_base.cc:1034] running on GCE node
W20240501 02:09:46.317479 1368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:46.317708 1360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:46.317953 1360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:46.319097 1360 hybrid_clock.cc:648] HybridClock initialized: now 1714529386319077 us; error 38 us; skew 500 ppm
May 01 02:09:46 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529386, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:46.321704 1360 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:09:46.322156 1360 webserver.cc:460] Webserver started at http://127.22.12.126:41931/ using document root <none> and password file <none>
I20240501 02:09:46.322351 1360 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:46.322387 1360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:46.322503 1360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:09:46 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529386, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:09:46.383193 1360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "aad64f19d9d641058d330d6428e7359b"
format_stamp: "Formatted at 2024-05-01 02:09:46 on dist-test-slave-9tkh"
server_key: "ce78e35d4948f75c25c097e959dd5ff5"
server_key_iv: "e020241926a865a9061166e063ca9e98"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:46.383567 1360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "aad64f19d9d641058d330d6428e7359b"
format_stamp: "Formatted at 2024-05-01 02:09:46 on dist-test-slave-9tkh"
server_key: "ce78e35d4948f75c25c097e959dd5ff5"
server_key_iv: "e020241926a865a9061166e063ca9e98"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:46.384793 1360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:46.400334 1375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:46.400518 1360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:09:46.400597 1360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "aad64f19d9d641058d330d6428e7359b"
format_stamp: "Formatted at 2024-05-01 02:09:46 on dist-test-slave-9tkh"
server_key: "ce78e35d4948f75c25c097e959dd5ff5"
server_key_iv: "e020241926a865a9061166e063ca9e98"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:46.400691 1360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:46.421605 1360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:46.424261 1360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:46.424397 1360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:46.429461 1360 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:44031
I20240501 02:09:46.429504 1427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:44031 every 8 connection(s)
I20240501 02:09:46.429910 1360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:09:46.430579 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:46.432711 1428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b: Bootstrap starting.
I20240501 02:09:46.433254 1428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:46.433495 1428 log.cc:824] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:46.434134 1428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b: No bootstrap required, opened a new log
I20240501 02:09:46.435973 1428 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad64f19d9d641058d330d6428e7359b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44031 } }
I20240501 02:09:46.436101 1428 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:46.436117 1428 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad64f19d9d641058d330d6428e7359b, State: Initialized, Role: FOLLOWER
I20240501 02:09:46.436232 1428 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad64f19d9d641058d330d6428e7359b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44031 } }
I20240501 02:09:46.436309 1428 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:09:46.436332 1428 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:09:46.436375 1428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:46.436887 1428 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad64f19d9d641058d330d6428e7359b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44031 } }
I20240501 02:09:46.436976 1428 leader_election.cc:302] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad64f19d9d641058d330d6428e7359b; no voters:
I20240501 02:09:46.437134 1428 leader_election.cc:288] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:09:46.437175 1431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:09:46.437330 1431 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [term 1 LEADER]: Becoming Leader. State: Replica: aad64f19d9d641058d330d6428e7359b, State: Running, Role: LEADER
I20240501 02:09:46.437381 1428 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:09:46.437471 1431 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad64f19d9d641058d330d6428e7359b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44031 } }
I20240501 02:09:46.437857 1431 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad64f19d9d641058d330d6428e7359b. Latest consensus state: current_term: 1 leader_uuid: "aad64f19d9d641058d330d6428e7359b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad64f19d9d641058d330d6428e7359b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44031 } } }
I20240501 02:09:46.437866 1433 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad64f19d9d641058d330d6428e7359b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad64f19d9d641058d330d6428e7359b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44031 } } }
I20240501 02:09:46.438046 1433 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:46.438234 1431 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b [sys.catalog]: This master's current role is: LEADER
I20240501 02:09:46.438548 1440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:09:46.438978 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 1360
I20240501 02:09:46.439026 1440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:09:46.439079 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:09:46.439268 22577 external_mini_cluster.cc:1428] Setting key e452c9776362dd760feabdc373f775df
I20240501 02:09:46.440186 1428 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240501 02:09:46.440749 1440 catalog_manager.cc:1347] Generated new cluster ID: d519fa3ef67e402cae17becbc59a610c
I20240501 02:09:46.442296 1440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:09:46 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:46.460234 1440 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:09:46.461011 1440 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:09:46.466668 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:46.439956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.22.12.126:44031)
0501 02:09:46.450764 (+ 10808us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:46.450788 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:46.450823 (+ 35us) server_negotiation.cc:407] Connection header received
0501 02:09:46.451039 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:46.451046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:46.451108 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:46.452396 (+ 1288us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:46.452653 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:46.454005 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:46.455183 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:46.457939 (+ 2756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:46.458083 (+ 144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:46.458091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:46.458091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:46.458109 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:46.461944 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:46.462337 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:46.462340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:46.462340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:46.462392 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:46.462593 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:46.462593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:46.462594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:46.462852 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:46.463476 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:46.463559 (+ 83us) server_negotiation.cc:299] Negotiation successful
0501 02:09:46.463696 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10712,"thread_start_us":82,"threads_started":1}
I20240501 02:09:46.475548 1440 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b: Generated new TSK 0
I20240501 02:09:46.475860 1440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:09:47.149230 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:36069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44031
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:47.243480 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:47.243849 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:47.243887 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:47.246186 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:47.246271 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:47.246311 1473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:09:47.248785 1473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.12.84:36069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:47.249070 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:47.249351 1473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:47.254665 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:47.254977 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:47.255410 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:47.255860 1473 server_base.cc:1034] running on GCE node
I20240501 02:09:47.256057 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:47.256306 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:47.257462 1473 hybrid_clock.cc:648] HybridClock initialized: now 1714529387257449 us; error 33 us; skew 500 ppm
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:47.260325 1473 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:09:47.260841 1473 webserver.cc:460] Webserver started at http://127.22.12.65:38521/ using document root <none> and password file <none>
I20240501 02:09:47.261055 1473 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:47.261107 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:47.261261 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:09:47.275352 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "9530528ae0c64f91bbc6f70699576591"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "089f2bedef6a05de1b5d8789818445dc"
server_key_iv: "3c76fe2e547b3c149ec35f8ee9868acd"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.275931 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "9530528ae0c64f91bbc6f70699576591"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "089f2bedef6a05de1b5d8789818445dc"
server_key_iv: "3c76fe2e547b3c149ec35f8ee9868acd"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.283978 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20240501 02:09:47.297871 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.298074 1473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:09:47.298163 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "9530528ae0c64f91bbc6f70699576591"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "089f2bedef6a05de1b5d8789818445dc"
server_key_iv: "3c76fe2e547b3c149ec35f8ee9868acd"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.298245 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:47.345202 1473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:47.347823 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:47.348021 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:47.348378 1473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:47.348807 1473 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:47.348888 1473 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.348977 1473 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:47.349043 1473 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.356719 1473 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:35085
I20240501 02:09:47.357228 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:09:47.363930 1603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:35085 every 8 connection(s)
I20240501 02:09:47.365473 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 1473
I20240501 02:09:47.365592 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:09:47.365775 22577 external_mini_cluster.cc:1428] Setting key 22b501c7c5402ff43177ada3abae6ff6
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:47.377517 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.364727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44031 (local address 127.22.12.65:48905)
0501 02:09:47.365117 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:47.365150 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:47.368709 (+ 3559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:47.369196 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:47.369201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:47.369469 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:47.369916 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.369923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.370980 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.370982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:47.371546 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.371549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.371696 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.372358 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:47.372382 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:47.374011 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:47.376013 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:47.376021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:47.376025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:47.376321 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:47.376516 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:47.376517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:47.376518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:47.376616 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:47.376886 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:47.376890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:47.377049 (+ 159us) client_negotiation.cc:769] Sending connection context
0501 02:09:47.377165 (+ 116us) client_negotiation.cc:240] Negotiation successful
0501 02:09:47.377303 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":91,"threads_started":1}
I20240501 02:09:47.378118 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.364726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:48905 (local address 127.22.12.126:44031)
0501 02:09:47.368950 (+ 4224us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.368952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.368964 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:09:47.368989 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.368990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.369003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.369060 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:47.370049 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.370875 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.371975 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.372152 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.374116 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:47.374122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:47.374122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:47.374130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:47.375905 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.376409 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.376411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.376411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.376436 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.376689 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.376690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.376690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.376808 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:47.376863 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.377900 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.378002 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4186,"thread_start_us":105,"threads_started":1}
I20240501 02:09:47.378906 1604 heartbeater.cc:347] Connected to a master server at 127.22.12.126:44031
I20240501 02:09:47.379010 1604 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:47.379244 1604 heartbeater.cc:511] Master 127.22.12.126:44031 requested a full tablet report, sending...
I20240501 02:09:47.379819 1392 ts_manager.cc:194] Registered new tserver with Master: 9530528ae0c64f91bbc6f70699576591 (127.22.12.65:35085)
I20240501 02:09:47.380725 1392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:48905
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:09:47.417555 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:36069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44031
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240501 02:09:47.534272 1610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:47.534628 1610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:47.534693 1610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:47.536984 1610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:47.537097 1610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:47.537170 1610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:09:47.540093 1610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.12.84:36069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:47.540364 1610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:47.540598 1610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:47.550220 1620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:47.550657 1618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:47.550989 1617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:47.551719 1610 server_base.cc:1034] running on GCE node
I20240501 02:09:47.551919 1610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:47.552160 1610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:47.553296 1610 hybrid_clock.cc:648] HybridClock initialized: now 1714529387553280 us; error 35 us; skew 500 ppm
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:47.556254 1610 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:09:47.556687 1610 webserver.cc:460] Webserver started at http://127.22.12.66:37327/ using document root <none> and password file <none>
I20240501 02:09:47.556851 1610 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:47.556890 1610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:47.556991 1610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:09:47.569058 1610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "2a4cb8c414234bf79b114b30054c5e68"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "6ba99002350c4455b799ebc0bfdd62b5"
server_key_iv: "133134b5b2ee7926075fc7abe309c491"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.569401 1610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "2a4cb8c414234bf79b114b30054c5e68"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "6ba99002350c4455b799ebc0bfdd62b5"
server_key_iv: "133134b5b2ee7926075fc7abe309c491"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.570869 1610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:09:47.584049 1628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.584383 1610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:09:47.584494 1610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "2a4cb8c414234bf79b114b30054c5e68"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "6ba99002350c4455b799ebc0bfdd62b5"
server_key_iv: "133134b5b2ee7926075fc7abe309c491"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.584586 1610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:47.609913 1610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:47.612126 1610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:47.612252 1610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:47.612490 1610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:47.612821 1610 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:47.612855 1610 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.612884 1610 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:47.612908 1610 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.619971 1610 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:39161
I20240501 02:09:47.620009 1741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:39161 every 8 connection(s)
I20240501 02:09:47.620484 1610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:47.629932 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 1610
I20240501 02:09:47.630055 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:09:47.630254 22577 external_mini_cluster.cc:1428] Setting key 4183ba281f266e7f9db3c1ea95f7489f
I20240501 02:09:47.631117 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.621475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:54975 (local address 127.22.12.126:44031)
0501 02:09:47.621517 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.621520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.621531 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:09:47.622702 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.622704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.622722 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.622793 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:47.623686 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.624472 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.625313 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.625490 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.627828 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:47.627833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:47.627833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:47.627841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:47.629600 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.630096 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.630097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.630098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.630126 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.630340 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.630341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.630342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.630459 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:47.630497 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.630902 (+ 405us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.631006 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:09:47.632104 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.621197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44031 (local address 127.22.12.66:54975)
0501 02:09:47.621388 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:47.621424 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:47.622489 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:47.622874 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:47.622880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:47.623140 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:47.623577 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.623585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.624575 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.624576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:47.625211 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.625215 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.625379 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.626032 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:47.626056 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:47.627718 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:47.629695 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:47.629702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:47.629706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:47.630008 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:47.630186 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:47.630186 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:47.630187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:47.630268 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:47.630571 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:47.630576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:47.630739 (+ 163us) client_negotiation.cc:769] Sending connection context
0501 02:09:47.631757 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0501 02:09:47.631901 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20240501 02:09:47.632752 1742 heartbeater.cc:347] Connected to a master server at 127.22.12.126:44031
I20240501 02:09:47.632849 1742 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:47.633076 1742 heartbeater.cc:511] Master 127.22.12.126:44031 requested a full tablet report, sending...
I20240501 02:09:47.633497 1392 ts_manager.cc:194] Registered new tserver with Master: 2a4cb8c414234bf79b114b30054c5e68 (127.22.12.66:39161)
I20240501 02:09:47.634003 1392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:54975
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:09:47.666167 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:36069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44031
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:09:47.759246 1747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:09:47.759528 1747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:09:47.759562 1747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:09:47.761431 1747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:09:47.761498 1747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:09:47.761533 1747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:09:47.763886 1747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.12.84:36069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:09:47.764163 1747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:09:47.764387 1747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:09:47.769049 1753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:47.769477 1752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:09:47.770030 1755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:09:47.770498 1747 server_base.cc:1034] running on GCE node
I20240501 02:09:47.770690 1747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:09:47.770892 1747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:09:47.772018 1747 hybrid_clock.cc:648] HybridClock initialized: now 1714529387772006 us; error 30 us; skew 500 ppm
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:09:47.774382 1747 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:09:47.774809 1747 webserver.cc:460] Webserver started at http://127.22.12.67:41769/ using document root <none> and password file <none>
I20240501 02:09:47.774977 1747 fs_manager.cc:359] Metadata directory not provided
I20240501 02:09:47.775017 1747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:09:47.775120 1747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:09:47.801951 1747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "979075abc60ef830fcac6b6807e7ec44"
server_key_iv: "05df75db6a88df4178272840f1f261f9"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.802366 1747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "979075abc60ef830fcac6b6807e7ec44"
server_key_iv: "05df75db6a88df4178272840f1f261f9"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.803984 1747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:09:47.838271 1762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.838580 1747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 02:09:47.838673 1747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07"
format_stamp: "Formatted at 2024-05-01 02:09:47 on dist-test-slave-9tkh"
server_key: "979075abc60ef830fcac6b6807e7ec44"
server_key_iv: "05df75db6a88df4178272840f1f261f9"
server_key_version: "kuduclusterkey@0"
I20240501 02:09:47.838765 1747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:09:47.881354 1747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:09:47.884606 1747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:09:47.884805 1747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:09:47.885063 1747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:09:47.885913 1747 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:09:47.886008 1747 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.886081 1747 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:09:47.886137 1747 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:09:47.892693 1747 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:34725
I20240501 02:09:47.892805 1875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:34725 every 8 connection(s)
I20240501 02:09:47.893240 1747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:09:47.893384 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 1747
I20240501 02:09:47.893501 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:09:47.893698 22577 external_mini_cluster.cc:1428] Setting key bdba5f81ec24d21ad68641422dcdc66e
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:09:47.906733 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.894722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:38003 (local address 127.22.12.126:44031)
0501 02:09:47.894765 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.894767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.894968 (+ 201us) server_negotiation.cc:407] Connection header received
0501 02:09:47.896071 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.896073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.896085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.896148 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:47.897090 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.897845 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.898645 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.898850 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.901146 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:47.901151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:47.901152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:47.901160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:47.902966 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.903426 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.903427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.903427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.903454 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.903728 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.903729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.903729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.903859 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:47.903932 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.906492 (+ 2560us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.906628 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:09:47.906934 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.894486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:44031 (local address 127.22.12.67:38003)
0501 02:09:47.894837 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:47.894868 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:47.895937 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:47.896282 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:47.896287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:47.896546 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:09:47.896980 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.896986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.897962 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.897963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:47.898563 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.898566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.898712 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.899367 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:09:47.899392 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:09:47.901035 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:09:47.903047 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:47.903053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:47.903057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:47.903344 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:47.903522 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:47.903523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:09:47.903523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:09:47.903618 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:47.906080 (+ 2462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:09:47.906087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:09:47.906288 (+ 201us) client_negotiation.cc:769] Sending connection context
0501 02:09:47.906444 (+ 156us) client_negotiation.cc:240] Negotiation successful
0501 02:09:47.906642 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":90,"threads_started":1}
I20240501 02:09:47.907536 1876 heartbeater.cc:347] Connected to a master server at 127.22.12.126:44031
I20240501 02:09:47.907653 1876 heartbeater.cc:464] Registering TS with master...
I20240501 02:09:47.907893 1876 heartbeater.cc:511] Master 127.22.12.126:44031 requested a full tablet report, sending...
I20240501 02:09:47.908288 1392 ts_manager.cc:194] Registered new tserver with Master: e6b721a7fe7f4e6bb4ac2126a1834d07 (127.22.12.67:34725)
I20240501 02:09:47.908824 1392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:38003
I20240501 02:09:47.916460 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:09:47.924984 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.918138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.22.12.126:44031)
0501 02:09:47.918179 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.918181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.918190 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:09:47.918261 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.918262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.918272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.918320 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:09:47.919253 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.920028 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.920817 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.920982 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.922218 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:09:47.922222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:09:47.922222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:09:47.922228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:09:47.923945 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.924324 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.924325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.924326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.924349 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:47.924482 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:09:47.924482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:09:47.924483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:09:47.924593 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:47.924692 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.924795 (+ 103us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.924881 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:09:47.926923 1392 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:09:47.928890 1392 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:09:47.936970 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.932014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:39161 (local address 127.0.0.1:41094)
0501 02:09:47.932730 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:47.932770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:47.932856 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:47.933136 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:47.933139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:47.933141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:47.933397 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.933400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.935835 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.935836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:47.936665 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.936668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.936736 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.936791 (+ 55us) client_negotiation.cc:769] Sending connection context
0501 02:09:47.936844 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 02:09:47.936892 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":82,"threads_started":1}
I20240501 02:09:47.937817 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.932107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.22.12.66:39161)
0501 02:09:47.932597 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.932609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.932851 (+ 242us) server_negotiation.cc:407] Connection header received
0501 02:09:47.932938 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.932940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.932958 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.933052 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:47.933478 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.935290 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.936848 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.937583 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.937652 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.937692 (+ 40us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.937738 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20240501 02:09:47.938509 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.932430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35085 (local address 127.0.0.1:51850)
0501 02:09:47.933512 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:47.933521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:47.933567 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:47.934042 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:47.934043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:47.934045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:47.934249 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.934251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.935816 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.935817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:47.938301 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.938305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.938372 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.938375 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:09:47.938401 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:09:47.938441 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":65,"threads_started":1}
I20240501 02:09:47.939189 1657 tablet_service.cc:1467] Processing CreateTablet for tablet 398b8a2b4f334a96be4e42d9ae9f922a (DEFAULT_TABLE table=test-table [id=d28817c1059d46dbb2691c00b7344cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:09:47.939532 1657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398b8a2b4f334a96be4e42d9ae9f922a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:47.939966 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.932558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.22.12.65:35085)
0501 02:09:47.932671 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.932676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.933622 (+ 946us) server_negotiation.cc:407] Connection header received
0501 02:09:47.933665 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.933667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.933684 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.933764 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:47.934337 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.935714 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.938820 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.939742 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.939799 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.939836 (+ 37us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.939888 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
I20240501 02:09:47.940784 1538 tablet_service.cc:1467] Processing CreateTablet for tablet 398b8a2b4f334a96be4e42d9ae9f922a (DEFAULT_TABLE table=test-table [id=d28817c1059d46dbb2691c00b7344cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:09:47.941058 1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398b8a2b4f334a96be4e42d9ae9f922a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:47.944300 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.932279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:34725 (local address 127.0.0.1:58280)
0501 02:09:47.933016 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:47.933025 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:47.933074 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:47.937146 (+ 4072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:47.937148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:47.937151 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:47.937338 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.937340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.942990 (+ 5650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.942991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:47.944029 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:47.944033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.944134 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.944140 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:09:47.944169 (+ 29us) client_negotiation.cc:240] Negotiation successful
0501 02:09:47.944219 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":67,"threads_started":1}
I20240501 02:09:47.944716 1895 tablet_bootstrap.cc:492] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: Bootstrap starting.
I20240501 02:09:47.945317 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:47.932359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.22.12.67:34725)
0501 02:09:47.932690 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:47.932697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:47.933839 (+ 1142us) server_negotiation.cc:407] Connection header received
0501 02:09:47.933878 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:47.933880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:47.933897 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.933969 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:47.937435 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.938443 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:47.944540 (+ 6097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:47.945114 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:47.945165 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:47.945201 (+ 36us) server_negotiation.cc:299] Negotiation successful
0501 02:09:47.945241 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":75,"threads_started":1}
I20240501 02:09:47.945437 1895 tablet_bootstrap.cc:654] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:47.945844 1895 log.cc:824] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:47.946053 1804 tablet_service.cc:1467] Processing CreateTablet for tablet 398b8a2b4f334a96be4e42d9ae9f922a (DEFAULT_TABLE table=test-table [id=d28817c1059d46dbb2691c00b7344cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:09:47.946337 1804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398b8a2b4f334a96be4e42d9ae9f922a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:09:47.946760 1895 tablet_bootstrap.cc:492] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: No bootstrap required, opened a new log
I20240501 02:09:47.946817 1895 ts_tablet_manager.cc:1367] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:09:47.949019 1895 raft_consensus.cc:348] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:47.949155 1895 raft_consensus.cc:374] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:47.949178 1895 raft_consensus.cc:729] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9530528ae0c64f91bbc6f70699576591, State: Initialized, Role: FOLLOWER
I20240501 02:09:47.949271 1895 consensus_queue.cc:260] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:47.949502 1895 ts_tablet_manager.cc:1398] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 02:09:47.949882 1894 tablet_bootstrap.cc:492] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Bootstrap starting.
I20240501 02:09:47.950001 1604 heartbeater.cc:503] Master 127.22.12.126:44031 was elected leader, sending a full tablet report...
I20240501 02:09:47.950663 1894 tablet_bootstrap.cc:654] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:47.950870 1897 tablet_bootstrap.cc:492] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Bootstrap starting.
I20240501 02:09:47.951458 1894 log.cc:824] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:47.952209 1897 tablet_bootstrap.cc:654] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Neither blocks nor log segments found. Creating new log.
I20240501 02:09:47.952209 1894 tablet_bootstrap.cc:492] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: No bootstrap required, opened a new log
I20240501 02:09:47.952277 1894 ts_tablet_manager.cc:1367] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:09:47.952796 1897 log.cc:824] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Log is configured to *not* fsync() on all Append() calls
I20240501 02:09:47.953505 1897 tablet_bootstrap.cc:492] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: No bootstrap required, opened a new log
I20240501 02:09:47.953564 1897 ts_tablet_manager.cc:1367] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 02:09:47.954089 1894 raft_consensus.cc:348] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:47.954223 1894 raft_consensus.cc:374] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:47.954243 1894 raft_consensus.cc:729] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4cb8c414234bf79b114b30054c5e68, State: Initialized, Role: FOLLOWER
I20240501 02:09:47.954373 1894 consensus_queue.cc:260] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:47.954649 1894 ts_tablet_manager.cc:1398] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:09:47.954761 1742 heartbeater.cc:503] Master 127.22.12.126:44031 was elected leader, sending a full tablet report...
I20240501 02:09:47.955423 1897 raft_consensus.cc:348] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:47.955545 1897 raft_consensus.cc:374] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:09:47.955564 1897 raft_consensus.cc:729] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b721a7fe7f4e6bb4ac2126a1834d07, State: Initialized, Role: FOLLOWER
I20240501 02:09:47.955648 1897 consensus_queue.cc:260] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:47.955875 1897 ts_tablet_manager.cc:1398] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:09:47.956070 1876 heartbeater.cc:503] Master 127.22.12.126:44031 was elected leader, sending a full tablet report...
May 01 02:09:47 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
W20240501 02:09:48.115499 1605 tablet.cc:2369] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:09:48.121999 1743 tablet.cc:2369] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:09:48.144985 1877 tablet.cc:2369] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:09:48.203722 1902 raft_consensus.cc:482] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:09:48.203864 1902 raft_consensus.cc:504] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:48.204283 1902 leader_election.cc:288] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a4cb8c414234bf79b114b30054c5e68 (127.22.12.66:39161), 9530528ae0c64f91bbc6f70699576591 (127.22.12.65:35085)
I20240501 02:09:48.207110 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.204462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:39161 (local address 127.22.12.67:40955)
0501 02:09:48.204583 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:48.204596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:48.204641 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:48.204916 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:48.204916 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:48.204918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:48.205084 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:48.205085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.205987 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:48.205988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:48.206879 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:48.206881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.206963 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:48.206967 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:09:48.206996 (+ 29us) client_negotiation.cc:240] Negotiation successful
0501 02:09:48.207037 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20240501 02:09:48.208140 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.204715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:40955 (local address 127.22.12.66:39161)
0501 02:09:48.204757 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:48.204759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:48.204766 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:09:48.204794 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:48.204795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:48.204804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:48.204846 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:48.205159 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.205874 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:48.207503 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.207996 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:48.208008 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:48.208033 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:09:48.208070 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:09:48.208436 1683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398b8a2b4f334a96be4e42d9ae9f922a" candidate_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4cb8c414234bf79b114b30054c5e68" is_pre_election: true
I20240501 02:09:48.208596 1683 raft_consensus.cc:2454] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b721a7fe7f4e6bb4ac2126a1834d07 in term 0.
I20240501 02:09:48.208839 1764 leader_election.cc:302] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4cb8c414234bf79b114b30054c5e68, e6b721a7fe7f4e6bb4ac2126a1834d07; no voters:
I20240501 02:09:48.209199 1902 raft_consensus.cc:2781] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:09:48.209280 1902 raft_consensus.cc:482] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:09:48.209306 1902 raft_consensus.cc:3037] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:48.210402 1902 raft_consensus.cc:504] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:48.210561 1902 leader_election.cc:288] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [CANDIDATE]: Term 1 election: Requested vote from peers 2a4cb8c414234bf79b114b30054c5e68 (127.22.12.66:39161), 9530528ae0c64f91bbc6f70699576591 (127.22.12.65:35085)
I20240501 02:09:48.210830 1683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398b8a2b4f334a96be4e42d9ae9f922a" candidate_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4cb8c414234bf79b114b30054c5e68"
I20240501 02:09:48.210909 1683 raft_consensus.cc:3037] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:48.211803 1683 raft_consensus.cc:2454] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b721a7fe7f4e6bb4ac2126a1834d07 in term 1.
I20240501 02:09:48.211998 1764 leader_election.cc:302] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4cb8c414234bf79b114b30054c5e68, e6b721a7fe7f4e6bb4ac2126a1834d07; no voters:
I20240501 02:09:48.212214 1902 raft_consensus.cc:2781] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:09:48.212358 1902 raft_consensus.cc:686] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 1 LEADER]: Becoming Leader. State: Replica: e6b721a7fe7f4e6bb4ac2126a1834d07, State: Running, Role: LEADER
I20240501 02:09:48.212445 1902 consensus_queue.cc:237] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } }
I20240501 02:09:48.213303 1391 catalog_manager.cc:5484] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6b721a7fe7f4e6bb4ac2126a1834d07 (127.22.12.67). New cstate: current_term: 1 leader_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 } health_report { overall_health: UNKNOWN } } }
I20240501 02:09:48.215430 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.204795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:35531 (local address 127.22.12.65:35085)
0501 02:09:48.204831 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:48.204833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:48.205253 (+ 420us) server_negotiation.cc:407] Connection header received
0501 02:09:48.205339 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:48.205340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:48.205349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:48.205372 (+ 23us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:09:48.210476 (+ 5104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.211145 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:48.213235 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.213670 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:48.213681 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:48.215322 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0501 02:09:48.215376 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:09:48.215425 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.204488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35085 (local address 127.22.12.67:35531)
0501 02:09:48.205195 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:48.205204 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:09:48.205277 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:09:48.206765 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:09:48.206766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:09:48.206769 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:09:48.210350 (+ 3581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:48.210354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.211796 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:48.211797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:09:48.213125 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:09:48.213128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.215214 (+ 2086us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:48.215222 (+ 8us) client_negotiation.cc:769] Sending connection context
0501 02:09:48.215296 (+ 74us) client_negotiation.cc:240] Negotiation successful
0501 02:09:48.215344 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":99,"threads_started":1}
I20240501 02:09:48.215862 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398b8a2b4f334a96be4e42d9ae9f922a" candidate_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9530528ae0c64f91bbc6f70699576591"
I20240501 02:09:48.215982 1558 raft_consensus.cc:3037] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:09:48.216625 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398b8a2b4f334a96be4e42d9ae9f922a" candidate_uuid: "e6b721a7fe7f4e6bb4ac2126a1834d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9530528ae0c64f91bbc6f70699576591" is_pre_election: true
I20240501 02:09:48.216923 1558 raft_consensus.cc:2454] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b721a7fe7f4e6bb4ac2126a1834d07 in term 1.
I20240501 02:09:48.221545 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.219269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.22.12.67:34725)
0501 02:09:48.219324 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:09:48.219327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:09:48.219336 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:09:48.219379 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:09:48.219380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:09:48.219400 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:09:48.219488 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:09:48.219715 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.220231 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:48.220842 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:48.220977 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:48.221012 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:48.221122 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:48.221204 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 02:09:48.221445 (+ 241us) server_negotiation.cc:299] Negotiation successful
0501 02:09:48.221472 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:09:48.223683 1683 raft_consensus.cc:1261] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 1 FOLLOWER]: Refusing update from remote peer e6b721a7fe7f4e6bb4ac2126a1834d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:09:48.224097 1902 consensus_queue.cc:1035] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a4cb8c414234bf79b114b30054c5e68" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 39161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:09:48.225806 1558 raft_consensus.cc:1261] T 398b8a2b4f334a96be4e42d9ae9f922a P 9530528ae0c64f91bbc6f70699576591 [term 1 FOLLOWER]: Refusing update from remote peer e6b721a7fe7f4e6bb4ac2126a1834d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
May 01 02:09:48 dist-test-slave-9tkh krb5kdc[915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529387, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:09:48.227123 1905 consensus_queue.cc:1035] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9530528ae0c64f91bbc6f70699576591" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 35085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:09:48.228072 1908 mvcc.cc:205] Tried to move back new op lower bound from 7022712374160416768 to 7022712374118625280. Current Snapshot: MvccSnapshot[applied={T|T < 7022712374160416768}]
I20240501 02:09:48.230448 1391 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43648:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:09:48.230564 1391 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43648:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:09:48.231272 1391 catalog_manager.cc:5771] T 00000000000000000000000000000000 P aad64f19d9d641058d330d6428e7359b: Sending DeleteTablet for 3 replicas of tablet 398b8a2b4f334a96be4e42d9ae9f922a
I20240501 02:09:48.231709 1538 tablet_service.cc:1514] Processing DeleteTablet for tablet 398b8a2b4f334a96be4e42d9ae9f922a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:09:48 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:51850
I20240501 02:09:48.232199 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 1473
I20240501 02:09:48.235890 1804 tablet_service.cc:1514] Processing DeleteTablet for tablet 398b8a2b4f334a96be4e42d9ae9f922a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:09:48 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:58280
I20240501 02:09:48.236197 1918 tablet_replica.cc:330] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: stopping tablet replica
I20240501 02:09:48.236289 1918 raft_consensus.cc:2229] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:09:48.236368 1918 raft_consensus.cc:2258] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:09:48.235766 1657 tablet_service.cc:1514] Processing DeleteTablet for tablet 398b8a2b4f334a96be4e42d9ae9f922a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:09:48 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:41094
I20240501 02:09:48.238842 1909 mvcc.cc:205] Tried to move back new op lower bound from 7022712374160416768 to 7022712374118625280. Current Snapshot: MvccSnapshot[applied={T|T < 7022712374160416768}]
I20240501 02:09:48.245440 1918 ts_tablet_manager.cc:1875] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:09:48.247308 1919 tablet_replica.cc:330] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: stopping tablet replica
I20240501 02:09:48.247496 1918 ts_tablet_manager.cc:1888] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:09:48.247648 1918 log.cc:1196] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/398b8a2b4f334a96be4e42d9ae9f922a
I20240501 02:09:48.247987 1918 ts_tablet_manager.cc:1909] T 398b8a2b4f334a96be4e42d9ae9f922a P e6b721a7fe7f4e6bb4ac2126a1834d07: Deleting consensus metadata
I20240501 02:09:48.248467 1919 raft_consensus.cc:2229] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:09:48.248495 1378 catalog_manager.cc:4830] TS e6b721a7fe7f4e6bb4ac2126a1834d07 (127.22.12.67:34725): tablet 398b8a2b4f334a96be4e42d9ae9f922a (table test-table [id=d28817c1059d46dbb2691c00b7344cdb]) successfully deleted
I20240501 02:09:48.248733 1919 raft_consensus.cc:2258] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:09:48.249737 1919 ts_tablet_manager.cc:1875] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:09:48.251364 1919 ts_tablet_manager.cc:1888] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:09:48.251479 1919 log.cc:1196] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/398b8a2b4f334a96be4e42d9ae9f922a
I20240501 02:09:48.251884 1919 ts_tablet_manager.cc:1909] T 398b8a2b4f334a96be4e42d9ae9f922a P 2a4cb8c414234bf79b114b30054c5e68: Deleting consensus metadata
I20240501 02:09:48.252379 1377 catalog_manager.cc:4830] TS 2a4cb8c414234bf79b114b30054c5e68 (127.22.12.66:39161): tablet 398b8a2b4f334a96be4e42d9ae9f922a (table test-table [id=d28817c1059d46dbb2691c00b7344cdb]) successfully deleted
W20240501 02:09:48.257074 1765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:35085 (error 108)
W20240501 02:09:48.257771 1378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:35085 (error 108)
I20240501 02:09:48.258653 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.258516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35085 (local address 127.0.0.1:51864)
0501 02:09:48.258580 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:48.258616 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:35085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20240501 02:09:48.258669 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 1610
W20240501 02:09:48.260623 1378 catalog_manager.cc:4573] TS 9530528ae0c64f91bbc6f70699576591 (127.22.12.65:35085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 398b8a2b4f334a96be4e42d9ae9f922a: Network error: Client connection negotiation failed: client connection to 127.22.12.65:35085: connect: Connection refused (error 111)
I20240501 02:09:48.261137 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:09:48.260990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:35085 (local address 127.22.12.67:55257)
0501 02:09:48.261038 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:48.261100 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:35085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20240501 02:09:48.265830 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 1747
I20240501 02:09:48.270771 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 1360
I20240501 02:09:48.277514 22577 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240501 02:09:48.622083 22577 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240501 02:09:48.622202 22577 mini_ranger.cc:66] Stopping Ranger...
I20240501 02:09:48.956264 22577 mini_ranger.cc:68] Stopped Ranger
2024-05-01 02:09:48.956 UTC [945] LOG: received smart shutdown request
2024-05-01 02:09:48.959 UTC [945] LOG: background worker "logical replication launcher" (PID 954) exited with exit code 1
2024-05-01 02:09:48.960 UTC [949] LOG: shutting down
2024-05-01 02:09:49.118 UTC [945] LOG: database system is shut down
2024-05-01T02:09:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44938 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:49 dist-test-slave-9tkh krb5kdc[1932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:49 dist-test-slave-9tkh krb5kdc[1932](info): set up 2 sockets
May 01 02:09:49 dist-test-slave-9tkh krb5kdc[1932](info): commencing operation
krb5kdc: starting...
W20240501 02:09:50.110276 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:09:50 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:09:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/rangeradmin_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/rangeradmin_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/rangerlookup_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/rangerlookup_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
I20240501 02:09:50.262856 22577 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBKBEtG/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/postgres -l logfile start
2024-05-01 02:09:52.695 UTC [1962] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-01 02:09:52.695 UTC [1962] LOG: listening on IPv4 address "127.22.12.84", port 49173
2024-05-01 02:09:52.698 UTC [1962] LOG: listening on Unix socket "/tmp/.s.PGSQL.49173"
2024-05-01 02:09:52.718 UTC [1965] LOG: database system was shut down at 2024-05-01 02:09:51 UTC
2024-05-01 02:09:52.722 UTC [1962] LOG: database system is ready to accept connections
I20240501 02:09:53.538111 22577 mini_postgres.cc:96] Postgres bound to 49173
2024-05-01 02:09:53.543 UTC [1973] FATAL: database "slave" does not exist
127.22.12.84:49173 - accepting connections
I20240501 02:09:53.543969 22577 mini_ranger.cc:161] Starting Ranger...
I20240501 02:09:53.557613 22577 mini_ranger.cc:84] Created miniranger Postgres user
I20240501 02:09:54.001048 22577 mini_ranger.cc:87] Created ranger Postgres database
I20240501 02:09:54.001117 22577 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-admin
2024-05-01 02:09:54,210 [I] DB FLAVOR :POSTGRES
2024-05-01 02:09:54,210 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 02:09:54,210 [I] Checking connection..
2024-05-01 02:09:54,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-01 02:09:54,451 [I] Checking connection passed.
2024-05-01 02:09:54,451 [I] --------- Verifying version history table ---------
2024-05-01 02:09:54,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 02:09:54,690 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 02:09:54,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 02:09:54,940 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 02:09:54,940 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-01 02:09:54,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-01 02:09:55.172 UTC [2037] WARNING: there is no transaction in progress
2024-05-01 02:09:55,178 [I] create_dbversion_catalog.sql file imported successfully
2024-05-01 02:09:55,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 02:09:55,433 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-01 02:09:55,434 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 02:09:55,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-01 02:09:55,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-01 02:09:55,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tkh.c.gcp-upstream.internal','N') ;"
2024-05-01 02:09:56,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 02:09:56,415 [I] Table x_portal_user does not exist in database ranger
2024-05-01 02:09:56,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 02:09:56,659 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-01 02:09:56,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 02:09:56,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 02:09:57,139 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-01 02:09:57,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-01 02:09:58.102 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.119 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.135 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.151 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.167 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.185 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.241 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.248 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.263 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.274 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.285 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.299 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.310 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.317 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.325 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58.332 UTC [2145] WARNING: there is no transaction in progress
2024-05-01 02:09:58,987 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-01 02:09:58,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-01 02:09:59,232 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-01 02:09:59,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 02:09:59,491 [I] Table x_portal_user already exists in database 'ranger'
2024-05-01 02:09:59,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 02:09:59,752 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-01 02:09:59,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 02:10:00,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 02:10:00,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tkh.c.gcp-upstream.internal';"
2024-05-01 02:10:00,482 [I] CORE_DB_SCHEMA import status has been updated
2024-05-01 02:10:00,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-01 02:10:00,737 [I] DB_PATCHES have already been applied
I20240501 02:10:00.743281 22577 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBKBEtG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240501 02:10:00.743355 22577 mini_ranger.cc:193] Using host: 127.22.12.84
I20240501 02:10:00.746160 22577 mini_ranger.cc:227] Ranger admin URL: http://127.22.12.84:44305
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp] .....
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-admin/ews/webapp].
May 01, 2024 2:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240501 02:10:01.681095 22577 mini_ranger.cc:160] Time spent starting Ranger: real 8.137s user 0.002s sys 0.007s
May 01, 2024 2:10:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 01, 2024 2:10:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 01, 2024 2:10:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 01, 2024 2:10:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 01, 2024 2:10:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240501 02:10:23.725829 22577 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/rangerkms_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/rangerkms_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/HTTP_127.22.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 01 02:10:23 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529423, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240501 02:10:23.800453 22577 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240501 02:10:23.809345 22577 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240501 02:10:24.637503 22577 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240501 02:10:24.637571 22577 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms
I20240501 02:10:24.637583 22577 mini_ranger_kms.cc:227] Using postgres at 127.22.12.84:49173
2024-05-01 02:10:24,789 [I] DB FLAVOR :POSTGRES
2024-05-01 02:10:24,790 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 02:10:24,790 [I] Checking connection
2024-05-01 02:10:24,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-01 02:10:25,039 [I] connection success
2024-05-01 02:10:25,039 [I] --------- Verifying Ranger DB tables ---------
2024-05-01 02:10:25,039 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-01 02:10:25,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-01 02:10:25,294 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-01 02:10:25,294 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 02:10:25,294 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-01 02:10:25,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.12.84:49173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-01 02:10:25,583 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240501 02:10:26.056326 22577 mini_ranger_kms.cc:310] Created kms service
I20240501 02:10:26.272293 22577 mini_ranger_kms.cc:326] Created kudu user
I20240501 02:10:26.314440 22577 mini_ranger_kms.cc:343] Created rangerkms user
May 01 02:10:26 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529423, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:26.681221 22577 mini_ranger_kms.cc:384] Added ranger policy
I20240501 02:10:26.681440 22577 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBKBEtG/build/release/bin/postgresql.jar:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBKBEtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240501 02:10:26.681471 22577 mini_ranger_kms.cc:240] Using host: 127.22.12.84
I20240501 02:10:26.684367 22577 mini_ranger_kms.cc:276] Ranger KMS PID: 2325
I20240501 02:10:26.684466 22577 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.12.84:55779
May 01, 2024 2:10:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 2:10:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 2:10:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 01, 2024 2:10:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/ews/webapp] .....
May 01, 2024 2:10:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-kms/ews/webapp].
May 01, 2024 2:10:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240501 02:10:28.540446 22577 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.740s user 0.013s sys 0.122s
I20240501 02:10:28.540529 22577 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240501 02:10:28.540573 22577 mini_ranger_kms.cc:401] 127.22.12.84:55779/kms/v1/keys
May 01 02:10:28 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714529428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 01 02:10:29 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.12.84@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
May 01 02:10:29 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.12.84@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
May 01 02:10:31 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:10:31.827598 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41139
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41139
--ranger_config_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:31.931066 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:31.931270 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:31.931293 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:31.932685 2376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:31.932735 2376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:31.932747 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:31.932758 2376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:31.932768 2376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:31.934407 2376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.12.84:55779/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41139
--ranger_config_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:31.934671 2376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:31.934857 2376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:31.940253 2382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:31.940539 2381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:31.941028 2384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:31.941416 2376 server_base.cc:1034] running on GCE node
I20240501 02:10:31.941733 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:31.942045 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:31.943207 2376 hybrid_clock.cc:648] HybridClock initialized: now 1714529431943186 us; error 38 us; skew 500 ppm
May 01 02:10:31 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:31.950865 2376 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:31.951408 2376 webserver.cc:460] Webserver started at http://127.22.12.126:37061/ using document root <none> and password file <none>
I20240501 02:10:31.951615 2376 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:31.951659 2376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:31.951772 2376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:10:31 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:32.004477 2376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "cd835c3756be446dae5b6705d276e571"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "71841fde2480e7e05b534cb65aa872e8"
server_key_iv: "539162c4e3cc5a9a3d4cf4b8f315a62d"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:32.004855 2376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "cd835c3756be446dae5b6705d276e571"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "71841fde2480e7e05b534cb65aa872e8"
server_key_iv: "539162c4e3cc5a9a3d4cf4b8f315a62d"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:32.006083 2376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:10:32.017911 2391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:32.018157 2376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:32.018245 2376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "cd835c3756be446dae5b6705d276e571"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "71841fde2480e7e05b534cb65aa872e8"
server_key_iv: "539162c4e3cc5a9a3d4cf4b8f315a62d"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:32.018330 2376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:32.036470 2376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:32.038403 2376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:32.038511 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:32.042757 2376 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41139
I20240501 02:10:32.042804 2443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41139 every 8 connection(s)
I20240501 02:10:32.043123 2376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:32.043769 2444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:32.044055 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 2376
I20240501 02:10:32.044147 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:10:32.044307 22577 external_mini_cluster.cc:1428] Setting key 5bae35f40eaacdca7179669c708258c2
I20240501 02:10:32.046443 2444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Bootstrap starting.
I20240501 02:10:32.047097 2444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:32.047361 2444 log.cc:824] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:32.048034 2444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: No bootstrap required, opened a new log
I20240501 02:10:32.050033 2444 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
May 01 02:10:32 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:32.050170 2444 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:32.050189 2444 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd835c3756be446dae5b6705d276e571, State: Initialized, Role: FOLLOWER
I20240501 02:10:32.050308 2444 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:32.050374 2444 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:32.050406 2444 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:32.050451 2444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:32.051128 2444 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:32.051240 2444 leader_election.cc:302] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd835c3756be446dae5b6705d276e571; no voters:
I20240501 02:10:32.051393 2444 leader_election.cc:288] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:10:32.051520 2449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:32.051605 2444 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:32.051666 2449 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 LEADER]: Becoming Leader. State: Replica: cd835c3756be446dae5b6705d276e571, State: Running, Role: LEADER
I20240501 02:10:32.051757 2449 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:32.052093 2450 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd835c3756be446dae5b6705d276e571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } } }
I20240501 02:10:32.052173 2450 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:32.052971 2444 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240501 02:10:32.052090 2451 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd835c3756be446dae5b6705d276e571. Latest consensus state: current_term: 1 leader_uuid: "cd835c3756be446dae5b6705d276e571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } } }
I20240501 02:10:32.053336 2451 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:32.055295 2457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:32.056164 2457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:32.056501 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:32.044819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.22.12.126:41139)
0501 02:10:32.045016 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:32.045029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:32.045055 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:10:32.045572 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:32.045581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:32.045637 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:32.045796 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:32.046478 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:32.047584 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:32.048343 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:32.048587 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:32.050940 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:32.050948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:32.050949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:32.050966 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:32.053171 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:32.054562 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:32.054613 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:32.054614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:32.054953 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:32.055188 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:32.055189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:32.055189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:32.055854 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:32.056054 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:32.056155 (+ 101us) server_negotiation.cc:299] Negotiation successful
0501 02:10:32.056315 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20240501 02:10:32.058001 2457 catalog_manager.cc:1347] Generated new cluster ID: a0a4c6dc1cb54d258530d8cd36de8fbc
I20240501 02:10:32.058046 2457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:10:32.064116 2457 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:10:32.064877 2457 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:32.078378 2457 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Generated new TSK 0
I20240501 02:10:32.078645 2457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:10:32.709241 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41139
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:32.792971 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:32.793196 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:32.793229 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:32.794715 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:32.794768 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:32.794802 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:10:32.796422 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.12.84:55779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:32.796662 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:32.796869 2489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:32.801318 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:32.801752 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:32.802152 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:32.802624 2489 server_base.cc:1034] running on GCE node
I20240501 02:10:32.802810 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:32.803025 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:32.804162 2489 hybrid_clock.cc:648] HybridClock initialized: now 1714529432804155 us; error 44 us; skew 500 ppm
May 01 02:10:32 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529432, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:32.806636 2489 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:10:32.807026 2489 webserver.cc:460] Webserver started at http://127.22.12.65:36185/ using document root <none> and password file <none>
I20240501 02:10:32.807216 2489 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:32.807258 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:32.807353 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:10:32 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529432, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:32.820812 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "93adb91f7437404e9adcee284757092c"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "8b40f8a6227343ae8f35dc099b640c86"
server_key_iv: "5ac144288e91ca68bd70f092175ee95a"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:32.821331 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "93adb91f7437404e9adcee284757092c"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "8b40f8a6227343ae8f35dc099b640c86"
server_key_iv: "5ac144288e91ca68bd70f092175ee95a"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:32.826789 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20240501 02:10:32.840346 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:32.840544 2489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:32.840624 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "93adb91f7437404e9adcee284757092c"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "8b40f8a6227343ae8f35dc099b640c86"
server_key_iv: "5ac144288e91ca68bd70f092175ee95a"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:32.840708 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:32.864049 2489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:32.865914 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:32.866029 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:32.866225 2489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:32.866523 2489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:32.866595 2489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:32.866626 2489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:32.866649 2489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:32.872088 2489 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:41785
I20240501 02:10:32.872134 2618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:41785 every 8 connection(s)
I20240501 02:10:32.872469 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:10:32.874238 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 2489
I20240501 02:10:32.874341 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:10:32.874485 22577 external_mini_cluster.cc:1428] Setting key a16ad28c08596984a51ff623b14e26ac
May 01 02:10:32 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529432, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:32.882807 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:32.873233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:43617 (local address 127.22.12.126:41139)
0501 02:10:32.874616 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:32.874618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:32.874630 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:10:32.874664 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:32.874665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:32.874677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:32.874744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:32.875560 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:32.876277 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:32.877311 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:32.877518 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:32.879383 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:32.879389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:32.879389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:32.879396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:32.881176 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:32.881660 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:32.881662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:32.881662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:32.881687 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:32.881929 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:32.881930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:32.881930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:32.882053 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:32.882114 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:32.882602 (+ 488us) server_negotiation.cc:299] Negotiation successful
0501 02:10:32.882695 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":72,"threads_started":1}
I20240501 02:10:32.883419 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:32.873150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41139 (local address 127.22.12.65:43617)
0501 02:10:32.873319 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:32.873354 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:32.874378 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:32.874840 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:32.874845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:32.875106 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:32.875463 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:32.875468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:32.876383 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:32.876384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:32.876904 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:32.876906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:32.877027 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:32.877653 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:32.877678 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:32.879280 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:32.881277 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:32.881284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:32.881289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:32.881575 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:32.881764 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:32.881764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:32.881765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:32.881861 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:32.882141 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:32.882147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:32.882313 (+ 166us) client_negotiation.cc:769] Sending connection context
0501 02:10:32.882441 (+ 128us) client_negotiation.cc:240] Negotiation successful
0501 02:10:32.883215 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20240501 02:10:32.884083 2619 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41139
I20240501 02:10:32.884198 2619 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:32.884425 2619 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:32.884994 2408 ts_manager.cc:194] Registered new tserver with Master: 93adb91f7437404e9adcee284757092c (127.22.12.65:41785)
I20240501 02:10:32.885948 2408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:43617
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:10:32.910068 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41139
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:33.017323 2626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:33.017596 2626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:33.017638 2626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:33.019887 2626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:33.019965 2626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:33.020015 2626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:10:33.022483 2626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.12.84:55779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:33.022760 2626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:33.023028 2626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:33.028599 2633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.028939 2634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.029322 2636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:33.029899 2626 server_base.cc:1034] running on GCE node
I20240501 02:10:33.030117 2626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:33.030371 2626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:33.031531 2626 hybrid_clock.cc:648] HybridClock initialized: now 1714529433031503 us; error 45 us; skew 500 ppm
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:33.034837 2626 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:10:33.035385 2626 webserver.cc:460] Webserver started at http://127.22.12.66:33813/ using document root <none> and password file <none>
I20240501 02:10:33.035619 2626 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:33.035674 2626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:33.035842 2626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:33.053663 2626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "be3b2c3d277d47e3994d5e464f378cb7"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "8a6d1314c8d6499febac90b438a298fe"
server_key_iv: "55bf79d924ff33d68d4d82a4c0b50cd6"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.053993 2626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "be3b2c3d277d47e3994d5e464f378cb7"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "8a6d1314c8d6499febac90b438a298fe"
server_key_iv: "55bf79d924ff33d68d4d82a4c0b50cd6"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.055752 2626 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 02:10:33.068519 2643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.068733 2626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20240501 02:10:33.068815 2626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "be3b2c3d277d47e3994d5e464f378cb7"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "8a6d1314c8d6499febac90b438a298fe"
server_key_iv: "55bf79d924ff33d68d4d82a4c0b50cd6"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.068897 2626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:33.085948 2626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:33.087975 2626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:33.088104 2626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:33.088307 2626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:33.088636 2626 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:33.088672 2626 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.088701 2626 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:33.088738 2626 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.095562 2626 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:41911
I20240501 02:10:33.095590 2756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:41911 every 8 connection(s)
I20240501 02:10:33.095965 2626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:10:33.096154 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 2626
I20240501 02:10:33.096256 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:10:33.096418 22577 external_mini_cluster.cc:1428] Setting key a047393ee2fc63b5c186ba9e1288b2d4
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
I20240501 02:10:33.111203 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.097093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:42465 (local address 127.22.12.126:41139)
0501 02:10:33.097158 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:33.097160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:33.098778 (+ 1618us) server_negotiation.cc:407] Connection header received
0501 02:10:33.099725 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:33.099726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:33.099738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:33.099790 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:33.100437 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.100960 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.101486 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.101648 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.106615 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:33.106621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:33.106622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:33.106631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:33.109594 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.110139 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.110140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.110141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.110170 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.110400 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.110401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.110402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.110561 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:33.110606 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:33.110991 (+ 385us) server_negotiation.cc:299] Negotiation successful
0501 02:10:33.111097 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:10:33.111717 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.097091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41139 (local address 127.22.12.66:42465)
0501 02:10:33.098651 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:33.098687 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:33.099600 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:33.099858 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:33.099862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:33.100055 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:33.100338 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.100343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.101042 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.101043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:33.101411 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.101413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.101547 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.102088 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:33.102113 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:33.106493 (+ 4380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:33.109718 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.109726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.109730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.110040 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.110238 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.110239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.110240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.110333 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.110659 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:33.110665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:33.110832 (+ 167us) client_negotiation.cc:769] Sending connection context
0501 02:10:33.111364 (+ 532us) client_negotiation.cc:240] Negotiation successful
0501 02:10:33.111502 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":75,"threads_started":1}
I20240501 02:10:33.112154 2757 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41139
I20240501 02:10:33.112255 2757 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.112478 2757 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:33.112862 2408 ts_manager.cc:194] Registered new tserver with Master: be3b2c3d277d47e3994d5e464f378cb7 (127.22.12.66:41911)
I20240501 02:10:33.113384 2408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:42465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:10:33.129319 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41139
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:33.247396 2763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:33.247737 2763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:33.247803 2763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:33.250020 2763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:33.250154 2763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:33.250238 2763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:10:33.252950 2763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.12.84:55779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:33.253206 2763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:33.253458 2763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:33.260164 2768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.260510 2771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.260911 2769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:33.261337 2763 server_base.cc:1034] running on GCE node
I20240501 02:10:33.261548 2763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:33.261814 2763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:33.265087 2763 hybrid_clock.cc:648] HybridClock initialized: now 1714529433265060 us; error 46 us; skew 500 ppm
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:33.268246 2763 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:10:33.268733 2763 webserver.cc:460] Webserver started at http://127.22.12.67:35141/ using document root <none> and password file <none>
I20240501 02:10:33.268921 2763 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:33.268965 2763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:33.269116 2763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:33.282465 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "446c0731c59640518fad4fadc27a0abd"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "e1754695eb5ee6b5405d720cd36b77e4"
server_key_iv: "f99754e3cb49a9cbe45218a86368e121"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.282900 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "446c0731c59640518fad4fadc27a0abd"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "e1754695eb5ee6b5405d720cd36b77e4"
server_key_iv: "f99754e3cb49a9cbe45218a86368e121"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.284508 2763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:33.300912 2778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.301187 2763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:33.301270 2763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "446c0731c59640518fad4fadc27a0abd"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "e1754695eb5ee6b5405d720cd36b77e4"
server_key_iv: "f99754e3cb49a9cbe45218a86368e121"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.301352 2763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:33.323662 2763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:33.325569 2763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:33.325685 2763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:33.325879 2763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:33.326225 2763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:33.326260 2763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.326290 2763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:33.326329 2763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.333362 2763 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:46071
I20240501 02:10:33.333846 2763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:10:33.334568 2891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:46071 every 8 connection(s)
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:33.340883 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 2763
I20240501 02:10:33.340996 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:10:33.341207 22577 external_mini_cluster.cc:1428] Setting key cb5f6cbfc174cc9f6a775826f9415dce
I20240501 02:10:33.343300 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.334659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:45385 (local address 127.22.12.126:41139)
0501 02:10:33.334707 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:33.334709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:33.335014 (+ 305us) server_negotiation.cc:407] Connection header received
0501 02:10:33.335852 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:33.335854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:33.335867 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:33.335918 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:33.336573 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.337119 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.337723 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.337884 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.339750 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:33.339756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:33.339756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:33.339763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:33.341441 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.342195 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.342197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.342197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.342225 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.342435 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.342435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.342436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.342596 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:33.342650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:33.343074 (+ 424us) server_negotiation.cc:299] Negotiation successful
0501 02:10:33.343180 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:10:33.344286 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.334624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41139 (local address 127.22.12.67:45385)
0501 02:10:33.334896 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:33.334921 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:33.335729 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:33.335982 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:33.335986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:33.336187 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:33.336473 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.336478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.337234 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.337234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:33.337647 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.337649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.337791 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.338332 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:33.338350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:33.339636 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:33.341810 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.341817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.341821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.342108 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.342286 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.342287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.342287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.342375 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.342695 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:33.342700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:33.342909 (+ 209us) client_negotiation.cc:769] Sending connection context
0501 02:10:33.343462 (+ 553us) client_negotiation.cc:240] Negotiation successful
0501 02:10:33.343602 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
I20240501 02:10:33.344964 2892 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41139
I20240501 02:10:33.345101 2892 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.345369 2892 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:33.345858 2408 ts_manager.cc:194] Registered new tserver with Master: 446c0731c59640518fad4fadc27a0abd (127.22.12.67:46071)
I20240501 02:10:33.346648 2408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:45385
I20240501 02:10:33.354957 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:10:33.355562 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 2489
I20240501 02:10:33.362614 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 2626
I20240501 02:10:33.369657 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 2763
I20240501 02:10:33.374328 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 2376
I20240501 02:10:33.379207 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41139
--webserver_interface=127.22.12.126
--webserver_port=37061
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41139
--ranger_config_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:33.458979 2895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:33.459182 2895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:33.459211 2895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:33.460541 2895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:33.460584 2895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:33.460602 2895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:33.460618 2895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:33.460631 2895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:33.462414 2895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41139
--ranger_config_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:33.462710 2895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:33.462886 2895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:33.467219 2900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.467648 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.468070 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:33.468560 2895 server_base.cc:1034] running on GCE node
I20240501 02:10:33.468775 2895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:33.468999 2895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:33.470135 2895 hybrid_clock.cc:648] HybridClock initialized: now 1714529433470113 us; error 37 us; skew 500 ppm
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:33.472537 2895 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:33.472934 2895 webserver.cc:460] Webserver started at http://127.22.12.126:37061/ using document root <none> and password file <none>
I20240501 02:10:33.473085 2895 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:33.473126 2895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:33.517591 2895 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:33.518275 2911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.518450 2895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:33.518556 2895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "cd835c3756be446dae5b6705d276e571"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "71841fde2480e7e05b534cb65aa872e8"
server_key_iv: "539162c4e3cc5a9a3d4cf4b8f315a62d"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.518877 2895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:33.548808 2895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:33.550766 2895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:33.550865 2895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:33.554760 2895 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41139
I20240501 02:10:33.554811 2963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41139 every 8 connection(s)
I20240501 02:10:33.555117 2895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:33.556208 2964 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:10:33.556818 2964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Bootstrap starting.
I20240501 02:10:33.564853 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 2895
I20240501 02:10:33.565124 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:41785
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:41139
--webserver_port=36185
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:33.565562 2964 log.cc:824] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:33.567608 2964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:10:33.567906 2964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Bootstrap complete.
I20240501 02:10:33.571097 2964 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:33.571393 2964 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd835c3756be446dae5b6705d276e571, State: Initialized, Role: FOLLOWER
I20240501 02:10:33.571549 2964 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:33.571624 2964 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:33.571656 2964 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:33.571707 2964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:10:33.572965 2964 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:33.573096 2964 leader_election.cc:302] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd835c3756be446dae5b6705d276e571; no voters:
I20240501 02:10:33.573272 2964 leader_election.cc:288] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 02:10:33.573333 2968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 02:10:33.573545 2968 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [term 2 LEADER]: Becoming Leader. State: Replica: cd835c3756be446dae5b6705d276e571, State: Running, Role: LEADER
I20240501 02:10:33.573563 2964 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:33.573698 2968 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } }
I20240501 02:10:33.574061 2969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd835c3756be446dae5b6705d276e571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } } }
I20240501 02:10:33.574061 2970 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd835c3756be446dae5b6705d276e571. Latest consensus state: current_term: 2 leader_uuid: "cd835c3756be446dae5b6705d276e571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd835c3756be446dae5b6705d276e571" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41139 } } }
I20240501 02:10:33.574177 2969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:33.574177 2970 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:33.574677 2976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:33.575135 2976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:33.575246 2976 catalog_manager.cc:1259] Loaded cluster ID: a0a4c6dc1cb54d258530d8cd36de8fbc
I20240501 02:10:33.575313 2976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:10:33.575529 2964 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240501 02:10:33.576354 2976 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:33.576516 2976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Loaded TSK: 0
I20240501 02:10:33.576711 2976 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 02:10:33.677220 2967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:33.677526 2967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:33.677598 2967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:33.679809 2967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:33.679908 2967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:33.679965 2967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:10:33.682438 2967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:41785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=36185
--tserver_master_addrs=127.22.12.126:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:33.682842 2967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:33.683097 2967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:33.688385 3004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.688802 3002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.689265 3001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:33.689714 2967 server_base.cc:1034] running on GCE node
I20240501 02:10:33.689910 2967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:33.690171 2967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:33.691331 2967 hybrid_clock.cc:648] HybridClock initialized: now 1714529433691315 us; error 35 us; skew 500 ppm
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:33.695055 2967 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:10:33.696085 2967 webserver.cc:460] Webserver started at http://127.22.12.65:36185/ using document root <none> and password file <none>
I20240501 02:10:33.696355 2967 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:33.696485 2967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:33.715618 2967 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20240501 02:10:33.718827 3012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.719066 2967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20240501 02:10:33.719149 2967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "93adb91f7437404e9adcee284757092c"
format_stamp: "Formatted at 2024-05-01 02:10:32 on dist-test-slave-9tkh"
server_key: "8b40f8a6227343ae8f35dc099b640c86"
server_key_iv: "5ac144288e91ca68bd70f092175ee95a"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.719554 2967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:33.743683 2967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:33.746042 2967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:33.746222 2967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:33.746462 2967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:33.747045 2967 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:33.747118 2967 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.747169 2967 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:33.747198 2967 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.753464 2967 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:41785
I20240501 02:10:33.753970 2967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:10:33.763432 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 2967
I20240501 02:10:33.763521 3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:41785 every 8 connection(s)
I20240501 02:10:33.763748 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:41911
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:41139
--webserver_port=33813
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:33.799266 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.767303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41139 (local address 127.22.12.65:59659)
0501 02:10:33.774404 (+ 7101us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:33.774451 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:33.775658 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:33.776495 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:33.776499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:33.776794 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:33.777202 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.777208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.778783 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.778784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:33.779358 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.779360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.779505 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.783057 (+ 3552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:33.783087 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:33.794942 (+ 11855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:33.797513 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.797523 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.797526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.797864 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.798114 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.798116 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.798116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.798200 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.798670 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:33.798676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:33.798812 (+ 136us) client_negotiation.cc:769] Sending connection context
0501 02:10:33.798918 (+ 106us) client_negotiation.cc:240] Negotiation successful
0501 02:10:33.799059 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7046,"thread_start_us":113,"threads_started":1}
I20240501 02:10:33.800499 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.767688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:59659 (local address 127.22.12.126:41139)
0501 02:10:33.775913 (+ 8225us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:33.775926 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:33.775950 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:10:33.776088 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:33.776094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:33.776147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:33.776284 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:33.777353 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.778660 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.782608 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.782886 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.795135 (+ 12249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:33.795150 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:33.795150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:33.795180 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:33.797383 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.797971 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.797976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.797977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.798037 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.798267 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.798268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.798269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.798513 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:33.798640 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:33.800192 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0501 02:10:33.800309 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8171,"thread_start_us":118,"threads_started":1}
I20240501 02:10:33.802949 3126 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41139
I20240501 02:10:33.803076 3126 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.803300 3126 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
W20240501 02:10:33.804261 3126 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:41139 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 02:10:33.804335 3126 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.804433 3126 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:33.804798 3126 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.804899 3126 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
W20240501 02:10:33.805197 3126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 02:10:33.882493 3128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:33.882798 3128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:33.882843 3128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:33.885082 3128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:33.885161 3128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:33.885201 3128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:10:33.887729 3128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=33813
--tserver_master_addrs=127.22.12.126:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:33.888000 3128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:33.888233 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:33.893736 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.894057 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:33.894654 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:33.895121 3128 server_base.cc:1034] running on GCE node
I20240501 02:10:33.895443 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:33.895715 3128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:33.898957 3128 hybrid_clock.cc:648] HybridClock initialized: now 1714529433898901 us; error 64 us; skew 500 ppm
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:33.902304 3128 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:10:33.902915 3128 webserver.cc:460] Webserver started at http://127.22.12.66:33813/ using document root <none> and password file <none>
I20240501 02:10:33.903139 3128 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:33.903209 3128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:33.921460 3128 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20240501 02:10:33.922231 3146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.922427 3128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:33.922505 3128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "be3b2c3d277d47e3994d5e464f378cb7"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "8a6d1314c8d6499febac90b438a298fe"
server_key_iv: "55bf79d924ff33d68d4d82a4c0b50cd6"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:33.922940 3128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:33.942817 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:33.944988 3128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:33.945127 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:33.945362 3128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:33.945760 3128 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:33.945814 3128 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.945853 3128 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:33.945878 3128 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:33.952920 3128 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:41911
I20240501 02:10:33.953346 3128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:10:33.955010 3259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:41911 every 8 connection(s)
May 01 02:10:33 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:33.962836 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3128
I20240501 02:10:33.963270 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.12.84:55779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:46071
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:41139
--webserver_port=35141
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:33.964334 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.954094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:35555 (local address 127.22.12.126:41139)
0501 02:10:33.954142 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:33.954144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:33.954424 (+ 280us) server_negotiation.cc:407] Connection header received
0501 02:10:33.955497 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:33.955498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:33.955508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:33.955553 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:33.956411 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.957253 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.958062 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.958248 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.960193 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:33.960199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:33.960199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:33.960207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:33.961815 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.962294 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.962295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.962295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.962318 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:33.962599 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:33.962600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:33.962600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:33.962722 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:33.962779 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:33.964149 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0501 02:10:33.964241 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:10:33.964439 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:33.954097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41139 (local address 127.22.12.66:35555)
0501 02:10:33.954270 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:33.954304 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:33.955368 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:33.955617 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:33.955622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:33.955880 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:33.956305 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.956311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.957362 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:33.957363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:33.957958 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:33.957961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:33.958105 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:33.958675 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:33.958703 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:33.960085 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:33.961914 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.961920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.961923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.962214 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.962386 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:33.962387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:33.962387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:33.962479 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:33.962789 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:33.962795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:33.963978 (+ 1183us) client_negotiation.cc:769] Sending connection context
0501 02:10:33.964104 (+ 126us) client_negotiation.cc:240] Negotiation successful
0501 02:10:33.964246 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20240501 02:10:33.965265 3260 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41139
I20240501 02:10:33.965378 3260 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.965621 3260 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
W20240501 02:10:33.966275 3260 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:41139 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 02:10:33.966343 3260 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.966439 3260 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:33.966759 3260 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:33.966863 3260 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
W20240501 02:10:33.967125 3260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 02:10:34.083037 3263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:34.083300 3263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:34.083335 3263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:34.085418 3263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:34.085494 3263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:34.085542 3263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:10:34.087994 3263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.12.84:55779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:46071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=35141
--tserver_master_addrs=127.22.12.126:41139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:34.088258 3263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:34.088493 3263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:34.094002 3272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:34.094436 3270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:34.094879 3269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:34.095389 3263 server_base.cc:1034] running on GCE node
I20240501 02:10:34.095595 3263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:34.095837 3263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:34.097012 3263 hybrid_clock.cc:648] HybridClock initialized: now 1714529434096982 us; error 44 us; skew 500 ppm
May 01 02:10:34 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529434, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:34.100345 3263 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:10:34.100975 3263 webserver.cc:460] Webserver started at http://127.22.12.67:35141/ using document root <none> and password file <none>
I20240501 02:10:34.101187 3263 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:34.101255 3263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:10:34 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529434, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:34.120805 3263 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20240501 02:10:34.121526 3280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:34.121683 3263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:34.121767 3263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "446c0731c59640518fad4fadc27a0abd"
format_stamp: "Formatted at 2024-05-01 02:10:33 on dist-test-slave-9tkh"
server_key: "e1754695eb5ee6b5405d720cd36b77e4"
server_key_iv: "f99754e3cb49a9cbe45218a86368e121"
server_key_version: "kuduclusterkey@0"
I20240501 02:10:34.122180 3263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:34.143035 3263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:34.145524 3263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:34.145669 3263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:34.145932 3263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:34.146311 3263 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:34.146363 3263 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:34.146401 3263 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:34.146423 3263 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:34.153596 3263 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:46071
I20240501 02:10:34.154054 3263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:10:34.154551 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:46071 every 8 connection(s)
May 01 02:10:34 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529434, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:34.160121 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3263
I20240501 02:10:34.164283 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:34.155140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:43827 (local address 127.22.12.126:41139)
0501 02:10:34.155190 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:34.155192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:34.155204 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:10:34.156249 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:34.156251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:34.156265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:34.156334 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:34.157205 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:34.157735 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:34.158308 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:34.158468 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:34.160870 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:34.160873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:34.160874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:34.160880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:34.162505 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:34.163253 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:34.163254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:34.163254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:34.163286 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:34.163525 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:34.163526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:34.163527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:34.163636 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:34.163706 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:34.164078 (+ 372us) server_negotiation.cc:299] Negotiation successful
0501 02:10:34.164180 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:10:34.164391 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:34.154836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41139 (local address 127.22.12.67:43827)
0501 02:10:34.155000 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:34.155034 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:34.156073 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:34.156397 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:34.156402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:34.156659 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:34.157087 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:34.157094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:34.157834 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:34.157835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:34.158213 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:34.158214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:34.158346 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:34.158935 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:34.158958 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:34.160761 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:34.162837 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:34.162843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:34.162847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:34.163152 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:34.163354 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:34.163355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:34.163356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:34.163451 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:34.163724 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:34.163730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:34.163903 (+ 173us) client_negotiation.cc:769] Sending connection context
0501 02:10:34.164049 (+ 146us) client_negotiation.cc:240] Negotiation successful
0501 02:10:34.164184 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20240501 02:10:34.165067 3394 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41139
I20240501 02:10:34.165239 3394 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:34.165506 3394 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
W20240501 02:10:34.166414 3394 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:41139 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 02:10:34.166577 3394 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:34.166725 3394 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:34.167096 3394 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:34.167214 3394 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
W20240501 02:10:34.167672 3394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 02:10:34.169917 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:34.160494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.22.12.126:41139)
0501 02:10:34.160593 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:34.160595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:34.160732 (+ 137us) server_negotiation.cc:407] Connection header received
0501 02:10:34.160793 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:34.160794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:34.160805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:34.160857 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:34.161928 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:34.163288 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:34.166060 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:34.166256 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:34.167341 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:34.167345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:34.167346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:34.167352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:34.168880 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:34.169180 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:34.169181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:34.169181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:34.169203 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:34.169344 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:34.169345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:34.169345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:34.169433 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:34.169575 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:34.169643 (+ 68us) server_negotiation.cc:299] Negotiation successful
0501 02:10:34.169736 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":45,"threads_started":1}
May 01 02:10:34 dist-test-slave-9tkh krb5kdc[1932](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714529434, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:34.805393 3126 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:34.805617 3126 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:34.806118 2928 ts_manager.cc:194] Registered new tserver with Master: 93adb91f7437404e9adcee284757092c (127.22.12.65:41785)
I20240501 02:10:34.807205 2928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:59659
I20240501 02:10:34.967312 3260 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:34.967522 3260 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:34.967912 2928 ts_manager.cc:194] Registered new tserver with Master: be3b2c3d277d47e3994d5e464f378cb7 (127.22.12.66:41911)
I20240501 02:10:34.968444 2928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:35555
May 01 02:10:34 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529434, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:35.167865 3394 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:35.168087 3394 heartbeater.cc:511] Master 127.22.12.126:41139 requested a full tablet report, sending...
I20240501 02:10:35.168521 2928 ts_manager.cc:194] Registered new tserver with Master: 446c0731c59640518fad4fadc27a0abd (127.22.12.67:46071)
I20240501 02:10:35.169057 2928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:43827
I20240501 02:10:35.171343 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:10:35.181931 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.175009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.22.12.126:41139)
0501 02:10:35.175349 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.175352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.175366 (+ 14us) server_negotiation.cc:407] Connection header received
0501 02:10:35.175391 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.175392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.175402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.176216 (+ 814us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:35.176272 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.177053 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.178222 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.178450 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.179423 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:35.179429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:35.179430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:35.179437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:35.180911 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:35.181243 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:35.181244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:35.181245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:35.181268 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:35.181408 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:35.181409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:35.181410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:35.181516 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:35.181604 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.181725 (+ 121us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.181825 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":90,"threads_started":1}
I20240501 02:10:35.183961 2928 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:10:35.185523 2928 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:10:35.194375 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.188889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.22.12.65:41785)
0501 02:10:35.189407 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.189413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.189757 (+ 344us) server_negotiation.cc:407] Connection header received
0501 02:10:35.189803 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.189805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.189823 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.189893 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:35.190338 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.191923 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.193624 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.194138 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.194198 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.194238 (+ 40us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.194279 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":56,"threads_started":1}
I20240501 02:10:35.194576 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.188694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:41785 (local address 127.0.0.1:44684)
0501 02:10:35.189663 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.189671 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:35.189710 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:35.189990 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:35.189993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:35.189995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:35.190253 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.190255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.192023 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.192024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:35.192967 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.192970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.193039 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.193091 (+ 52us) client_negotiation.cc:769] Sending connection context
0501 02:10:35.193142 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:10:35.193194 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":62,"threads_started":1}
I20240501 02:10:35.195221 3043 tablet_service.cc:1467] Processing CreateTablet for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec (DEFAULT_TABLE table=test-table [id=3df39f75194d445f80236e228a17977a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:35.195538 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:35.203500 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.188427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:46071 (local address 127.0.0.1:37188)
0501 02:10:35.189446 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.189474 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:35.189559 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:35.200212 (+ 10653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:35.200214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:35.200217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:35.200458 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.200460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.201953 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.201954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:35.203270 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.203274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.203327 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.203331 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:10:35.203356 (+ 25us) client_negotiation.cc:240] Negotiation successful
0501 02:10:35.203400 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":90,"threads_started":1}
I20240501 02:10:35.204561 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.188540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.22.12.67:46071)
0501 02:10:35.200026 (+ 11486us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.200031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.200041 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:10:35.200081 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.200082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.200115 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.200531 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:35.200605 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.201787 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.203769 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.204299 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.204360 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.204399 (+ 39us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.204435 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11451,"thread_start_us":58,"threads_started":1}
I20240501 02:10:35.204857 3427 tablet_bootstrap.cc:492] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: Bootstrap starting.
I20240501 02:10:35.205261 3328 tablet_service.cc:1467] Processing CreateTablet for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec (DEFAULT_TABLE table=test-table [id=3df39f75194d445f80236e228a17977a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:35.205554 3427 tablet_bootstrap.cc:654] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:35.205591 3328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:35.205857 3427 log.cc:824] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:35.209878 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.195492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:41911 (local address 127.0.0.1:38118)
0501 02:10:35.199862 (+ 4370us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.199874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:35.199916 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:35.206882 (+ 6966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:35.206884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:35.206892 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:35.207082 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.207085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.208770 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.208772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:35.209658 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.209673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.209728 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.209733 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:10:35.209756 (+ 23us) client_negotiation.cc:240] Negotiation successful
0501 02:10:35.209799 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4354,"thread_start_us":109,"threads_started":1}
I20240501 02:10:35.210929 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.202610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.22.12.66:41911)
0501 02:10:35.206224 (+ 3614us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.206228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.206242 (+ 14us) server_negotiation.cc:407] Connection header received
0501 02:10:35.206287 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.206289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.206321 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.207177 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:35.207258 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.208319 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.210132 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.210688 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.210746 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.210788 (+ 42us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.210834 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3569,"thread_start_us":83,"threads_started":1}
I20240501 02:10:35.211594 3186 tablet_service.cc:1467] Processing CreateTablet for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec (DEFAULT_TABLE table=test-table [id=3df39f75194d445f80236e228a17977a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:35.211910 3186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:35.211982 3427 tablet_bootstrap.cc:492] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: No bootstrap required, opened a new log
I20240501 02:10:35.212065 3427 ts_tablet_manager.cc:1367] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 02:10:35.214254 3427 raft_consensus.cc:348] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.214411 3427 raft_consensus.cc:374] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:35.214437 3427 raft_consensus.cc:729] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93adb91f7437404e9adcee284757092c, State: Initialized, Role: FOLLOWER
I20240501 02:10:35.214593 3427 consensus_queue.cc:260] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.214895 3427 ts_tablet_manager.cc:1398] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:35.214996 3126 heartbeater.cc:503] Master 127.22.12.126:41139 was elected leader, sending a full tablet report...
I20240501 02:10:35.215458 3432 tablet_bootstrap.cc:492] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: Bootstrap starting.
I20240501 02:10:35.216060 3432 tablet_bootstrap.cc:654] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:35.216349 3432 log.cc:824] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:35.216766 3430 tablet_bootstrap.cc:492] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Bootstrap starting.
I20240501 02:10:35.217365 3430 tablet_bootstrap.cc:654] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:35.217659 3430 log.cc:824] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:35.218688 3432 tablet_bootstrap.cc:492] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: No bootstrap required, opened a new log
I20240501 02:10:35.218767 3432 ts_tablet_manager.cc:1367] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 02:10:35.220873 3432 raft_consensus.cc:348] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.221022 3432 raft_consensus.cc:374] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:35.221042 3432 raft_consensus.cc:729] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3b2c3d277d47e3994d5e464f378cb7, State: Initialized, Role: FOLLOWER
I20240501 02:10:35.221156 3432 consensus_queue.cc:260] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.221400 3432 ts_tablet_manager.cc:1398] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:35.221498 3260 heartbeater.cc:503] Master 127.22.12.126:41139 was elected leader, sending a full tablet report...
I20240501 02:10:35.222349 3430 tablet_bootstrap.cc:492] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: No bootstrap required, opened a new log
I20240501 02:10:35.222412 3430 ts_tablet_manager.cc:1367] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 02:10:35.224681 3430 raft_consensus.cc:348] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.224841 3430 raft_consensus.cc:374] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:35.224867 3430 raft_consensus.cc:729] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c0731c59640518fad4fadc27a0abd, State: Initialized, Role: FOLLOWER
I20240501 02:10:35.224990 3430 consensus_queue.cc:260] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.225281 3430 ts_tablet_manager.cc:1398] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:35.225330 3394 heartbeater.cc:503] Master 127.22.12.126:41139 was elected leader, sending a full tablet report...
I20240501 02:10:35.227337 3435 raft_consensus.cc:482] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:35.227443 3435 raft_consensus.cc:504] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.228487 3435 leader_election.cc:288] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446c0731c59640518fad4fadc27a0abd (127.22.12.67:46071), 93adb91f7437404e9adcee284757092c (127.22.12.65:41785)
I20240501 02:10:35.242596 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.227953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:51271 (local address 127.22.12.67:46071)
0501 02:10:35.227993 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.227996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.232389 (+ 4393us) server_negotiation.cc:407] Connection header received
0501 02:10:35.232504 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.232506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.232516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.232543 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:35.239904 (+ 7361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.240665 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.241851 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.242309 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.242318 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.242440 (+ 122us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.242477 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:10:35.243090 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.227884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:46071 (local address 127.22.12.66:51271)
0501 02:10:35.232278 (+ 4394us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.232295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:35.232425 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:35.232619 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:35.232621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:35.232623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:35.237804 (+ 5181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.237810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.240787 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.240788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:35.241735 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.241740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.242353 (+ 613us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.242359 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:10:35.242946 (+ 587us) client_negotiation.cc:240] Negotiation successful
0501 02:10:35.243001 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4360,"thread_start_us":100,"threads_started":1}
I20240501 02:10:35.243829 3339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cb5f8191c94a7fbe1f6cb95ecb6eec" candidate_uuid: "be3b2c3d277d47e3994d5e464f378cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c0731c59640518fad4fadc27a0abd" is_pre_election: true
I20240501 02:10:35.244004 3339 raft_consensus.cc:2454] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be3b2c3d277d47e3994d5e464f378cb7 in term 0.
I20240501 02:10:35.244282 3150 leader_election.cc:302] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446c0731c59640518fad4fadc27a0abd, be3b2c3d277d47e3994d5e464f378cb7; no voters:
I20240501 02:10:35.244433 3435 raft_consensus.cc:2781] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:10:35.244518 3435 raft_consensus.cc:482] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:10:35.244542 3435 raft_consensus.cc:3037] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:35.245461 3435 raft_consensus.cc:504] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.245815 3339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cb5f8191c94a7fbe1f6cb95ecb6eec" candidate_uuid: "be3b2c3d277d47e3994d5e464f378cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c0731c59640518fad4fadc27a0abd"
I20240501 02:10:35.245893 3339 raft_consensus.cc:3037] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:35.246493 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.228154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:41785 (local address 127.22.12.66:32911)
0501 02:10:35.243725 (+ 15571us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.243736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:35.243815 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:35.244129 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:35.244130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:35.244133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:35.244322 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.244324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.245393 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.245394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:35.246277 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:35.246279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.246353 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.246358 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:10:35.246387 (+ 29us) client_negotiation.cc:240] Negotiation successful
0501 02:10:35.246423 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15555,"thread_start_us":134,"threads_started":1}
I20240501 02:10:35.246711 3339 raft_consensus.cc:2454] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be3b2c3d277d47e3994d5e464f378cb7 in term 1.
I20240501 02:10:35.246891 3150 leader_election.cc:302] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446c0731c59640518fad4fadc27a0abd, be3b2c3d277d47e3994d5e464f378cb7; no voters:
I20240501 02:10:35.247016 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.228424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:32911 (local address 127.22.12.65:41785)
0501 02:10:35.228466 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.228467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.243783 (+ 15316us) server_negotiation.cc:407] Connection header received
0501 02:10:35.243887 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.243888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.243898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.243923 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:35.244510 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.245214 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.246371 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.246885 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.246896 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.246921 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.246955 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:10:35.247313 3080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cb5f8191c94a7fbe1f6cb95ecb6eec" candidate_uuid: "be3b2c3d277d47e3994d5e464f378cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93adb91f7437404e9adcee284757092c" is_pre_election: true
I20240501 02:10:35.247462 3080 raft_consensus.cc:2454] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be3b2c3d277d47e3994d5e464f378cb7 in term 0.
I20240501 02:10:35.247717 3080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cb5f8191c94a7fbe1f6cb95ecb6eec" candidate_uuid: "be3b2c3d277d47e3994d5e464f378cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93adb91f7437404e9adcee284757092c"
I20240501 02:10:35.247742 3435 leader_election.cc:288] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [CANDIDATE]: Term 1 election: Requested vote from peers 446c0731c59640518fad4fadc27a0abd (127.22.12.67:46071), 93adb91f7437404e9adcee284757092c (127.22.12.65:41785)
I20240501 02:10:35.247779 3080 raft_consensus.cc:3037] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:35.247797 3435 raft_consensus.cc:2781] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:35.247828 3435 raft_consensus.cc:686] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 1 LEADER]: Becoming Leader. State: Replica: be3b2c3d277d47e3994d5e464f378cb7, State: Running, Role: LEADER
I20240501 02:10:35.247908 3435 consensus_queue.cc:237] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } }
I20240501 02:10:35.248628 3080 raft_consensus.cc:2454] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be3b2c3d277d47e3994d5e464f378cb7 in term 1.
I20240501 02:10:35.248783 2927 catalog_manager.cc:5484] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to be3b2c3d277d47e3994d5e464f378cb7 (127.22.12.66). New cstate: current_term: 1 leader_uuid: "be3b2c3d277d47e3994d5e464f378cb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be3b2c3d277d47e3994d5e464f378cb7" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 41911 } health_report { overall_health: HEALTHY } } }
I20240501 02:10:35.257100 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.254811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.22.12.66:41911)
0501 02:10:35.254849 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:35.254851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:35.254857 (+ 6us) server_negotiation.cc:407] Connection header received
0501 02:10:35.254901 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:35.254902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:35.254920 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:35.254986 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:10:35.255307 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.255809 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:35.256571 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:35.256729 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:35.256769 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:35.256857 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:35.256928 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:35.256999 (+ 71us) server_negotiation.cc:299] Negotiation successful
0501 02:10:35.257024 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:10:35.259552 3080 raft_consensus.cc:1261] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 1 FOLLOWER]: Refusing update from remote peer be3b2c3d277d47e3994d5e464f378cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:35.261056 3440 consensus_queue.cc:1035] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93adb91f7437404e9adcee284757092c" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:35.261694 3339 raft_consensus.cc:1261] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 1 FOLLOWER]: Refusing update from remote peer be3b2c3d277d47e3994d5e464f378cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:35.262794 3440 consensus_queue.cc:1035] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c0731c59640518fad4fadc27a0abd" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:35.266849 3446 mvcc.cc:205] Tried to move back new op lower bound from 7022712566820184064 to 7022712566776049664. Current Snapshot: MvccSnapshot[applied={T|T < 7022712566820184064}]
I20240501 02:10:35.266954 3443 mvcc.cc:205] Tried to move back new op lower bound from 7022712566820184064 to 7022712566776049664. Current Snapshot: MvccSnapshot[applied={T|T < 7022712566820184064}]
I20240501 02:10:35.267050 3444 mvcc.cc:205] Tried to move back new op lower bound from 7022712566820184064 to 7022712566776049664. Current Snapshot: MvccSnapshot[applied={T|T < 7022712566820184064}]
May 01 02:10:35 dist-test-slave-9tkh krb5kdc[1932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529434, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for HTTP/127.22.12.84@KRBTEST.COM
I20240501 02:10:35.273583 2927 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42856:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:35.273716 2927 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42856:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:35.277493 2927 catalog_manager.cc:5771] T 00000000000000000000000000000000 P cd835c3756be446dae5b6705d276e571: Sending DeleteTablet for 3 replicas of tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec
I20240501 02:10:35.278088 3328 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:35 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:37188
I20240501 02:10:35.278373 3186 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:35 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:38118
I20240501 02:10:35.278843 3043 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:35 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:44684
I20240501 02:10:35.279104 3454 tablet_replica.cc:330] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c: stopping tablet replica
I20240501 02:10:35.279160 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 2967
I20240501 02:10:35.279191 3454 raft_consensus.cc:2229] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 93adb91f7437404e9adcee284757092c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 02:10:35.289724 2912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:41785 (error 108)
I20240501 02:10:35.289920 3452 tablet_replica.cc:330] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: stopping tablet replica
I20240501 02:10:35.290040 3452 raft_consensus.cc:2229] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:35.290165 3452 raft_consensus.cc:2258] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:35.290354 3453 tablet_replica.cc:330] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: stopping tablet replica
I20240501 02:10:35.290454 3453 raft_consensus.cc:2229] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:10:35.290609 3453 raft_consensus.cc:2258] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:35.291129 3453 ts_tablet_manager.cc:1875] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P be3b2c3d277d47e3994d5e464f378cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:35.291877 3452 ts_tablet_manager.cc:1875] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:35.293735 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3128
I20240501 02:10:35.294414 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.293899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:41785 (local address 127.0.0.1:44690)
0501 02:10:35.294319 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.294373 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389}
W20240501 02:10:35.295094 2912 catalog_manager.cc:4573] TS 93adb91f7437404e9adcee284757092c (127.22.12.65:41785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45cb5f8191c94a7fbe1f6cb95ecb6eec: Network error: Client connection negotiation failed: client connection to 127.22.12.65:41785: connect: Connection refused (error 111)
I20240501 02:10:35.299376 3452 ts_tablet_manager.cc:1888] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:35.301034 3452 log.cc:1196] T 45cb5f8191c94a7fbe1f6cb95ecb6eec P 446c0731c59640518fad4fadc27a0abd: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/45cb5f8191c94a7fbe1f6cb95ecb6eec
I20240501 02:10:35.301046 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.300925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:41911 (local address 127.0.0.1:38138)
0501 02:10:35.300978 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.301003 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.66:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20240501 02:10:35.301107 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3263
I20240501 02:10:35.306771 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 2895
I20240501 02:10:35.306875 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:35.306746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:46071 (local address 127.0.0.1:37192)
0501 02:10:35.306791 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:35.306830 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.67:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20240501 02:10:35.315837 22577 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240501 02:10:35.660586 22577 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240501 02:10:35.660712 22577 mini_ranger.cc:66] Stopping Ranger...
I20240501 02:10:35.995296 22577 mini_ranger.cc:68] Stopped Ranger
2024-05-01 02:10:35.995 UTC [1962] LOG: received smart shutdown request
2024-05-01 02:10:35.999 UTC [1962] LOG: background worker "logical replication launcher" (PID 1971) exited with exit code 1
2024-05-01 02:10:35.999 UTC [1966] LOG: shutting down
2024-05-01 02:10:36.154 UTC [1962] LOG: database system is shut down
2024-05-01T02:10:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47026 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:10:36 dist-test-slave-9tkh krb5kdc[3465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:10:36 dist-test-slave-9tkh krb5kdc[3465](info): set up 2 sockets
May 01 02:10:36 dist-test-slave-9tkh krb5kdc[3465](info): commencing operation
krb5kdc: starting...
W20240501 02:10:37.142253 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.884s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:10:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:10:37.256996 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41169
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41169,127.22.12.125:40773,127.22.12.124:43145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:37.336719 3478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:37.336925 3478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:37.336947 3478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:37.338322 3478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:37.338375 3478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:37.338387 3478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:37.338398 3478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:37.338409 3478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:37.340143 3478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41169,127.22.12.125:40773,127.22.12.124:43145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:37.340389 3478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:37.340590 3478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:37.345160 3486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:37.345513 3483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:37.346084 3484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:37.346511 3478 server_base.cc:1034] running on GCE node
I20240501 02:10:37.346765 3478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:37.347012 3478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:37.348142 3478 hybrid_clock.cc:648] HybridClock initialized: now 1714529437348125 us; error 33 us; skew 500 ppm
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:37.354894 3478 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:37.355379 3478 webserver.cc:460] Webserver started at http://127.22.12.126:39451/ using document root <none> and password file <none>
I20240501 02:10:37.355566 3478 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:37.355609 3478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:37.355715 3478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:37.356722 3478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "3bed8cf7da6f402d8af87b223d1fa300"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "b2b9db866bce0668dd4a192bd822928d"
server_key_iv: "235e04ab21d36a728c47657447b4b692"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.357023 3478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "3bed8cf7da6f402d8af87b223d1fa300"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "b2b9db866bce0668dd4a192bd822928d"
server_key_iv: "235e04ab21d36a728c47657447b4b692"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.358304 3478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:10:37.359180 3493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:37.359422 3478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:10:37.359519 3478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "3bed8cf7da6f402d8af87b223d1fa300"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "b2b9db866bce0668dd4a192bd822928d"
server_key_iv: "235e04ab21d36a728c47657447b4b692"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.359588 3478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:37.368250 3478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:37.370307 3478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:37.370431 3478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:37.374296 3478 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41169
I20240501 02:10:37.374333 3545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41169 every 8 connection(s)
I20240501 02:10:37.374680 3478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:37.375417 3546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:37.377228 3546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.381651 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3478
I20240501 02:10:37.381774 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:10:37.381951 22577 external_mini_cluster.cc:1428] Setting key 9893f1ac41e42c42f7603301f208b8a7
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:37.386054 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.377867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41169 (local address 127.0.0.1:40610)
0501 02:10:37.378037 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.378061 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.378483 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.378744 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.378746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.379044 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.379279 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.379284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.380114 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.380115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.380496 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.380497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.380608 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.381202 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.381215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.382729 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.384750 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.384755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.384758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.384925 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.385137 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.385137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.385137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.385180 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.385577 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.385582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.385643 (+ 61us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.385740 (+ 97us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.385858 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20240501 02:10:37.386062 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.378036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.22.12.126:41169)
0501 02:10:37.378163 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.378169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.378191 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:10:37.378629 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.378631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.378649 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.378726 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.379359 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.380038 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.380566 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.380706 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.382823 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.382833 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.382834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.382839 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.384645 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.385014 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.385017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.385017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.385073 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.385234 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.385235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.385236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.385503 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.385602 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.385790 (+ 188us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.385931 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":102,"thread_start_us":43,"threads_started":1}
I20240501 02:10:37.386863 3546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.387217 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.387096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46004)
0501 02:10:37.387144 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.387188 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240501 02:10:37.387326 3497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
I20240501 02:10:37.387709 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.387619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46020)
0501 02:10:37.387663 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.387687 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 02:10:37.387804 3546 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:40773: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.22.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab.
Entry for principal kudu/127.22.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab.
I20240501 02:10:37.405759 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:40773
--webserver_interface=127.22.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab
--principal=kudu/127.22.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41169,127.22.12.125:40773,127.22.12.124:43145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:37.437017 3546 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } attempt: 1
I20240501 02:10:37.437566 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.437389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46034)
0501 02:10:37.437467 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.437521 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 02:10:37.438004 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.437898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46048)
0501 02:10:37.437943 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.437976 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240501 02:10:37.438148 3546 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:40773: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
W20240501 02:10:37.484982 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:37.485184 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:37.485205 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:37.486626 3553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:37.486688 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:37.486701 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:37.486712 3553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:37.486722 3553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:37.488525 3553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41169,127.22.12.125:40773,127.22.12.124:43145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab
--principal=kudu/127.22.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:37.488765 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:37.488971 3553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:37.493148 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:37.493678 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:37.494179 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:37.494686 3553 server_base.cc:1034] running on GCE node
I20240501 02:10:37.494935 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:37.495167 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:37.496304 3553 hybrid_clock.cc:648] HybridClock initialized: now 1714529437496292 us; error 30 us; skew 500 ppm
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:37.498698 3553 init.cc:377] Logged in from keytab as kudu/127.22.12.125@KRBTEST.COM (short username kudu)
I20240501 02:10:37.499107 3553 webserver.cc:460] Webserver started at http://127.22.12.125:45461/ using document root <none> and password file <none>
I20240501 02:10:37.499285 3553 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:37.499326 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:37.499429 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:37.500447 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data/instance:
uuid: "0e1f5eb5e6924a76a0dfde22c63a9963"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "90aee719a18aff42fde1b4ce3851d8b4"
server_key_iv: "e20dd59e17bcdf81f9bdbfa97f5460b2"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.500739 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal/instance:
uuid: "0e1f5eb5e6924a76a0dfde22c63a9963"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "90aee719a18aff42fde1b4ce3851d8b4"
server_key_iv: "e20dd59e17bcdf81f9bdbfa97f5460b2"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.501866 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:10:37.502511 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:37.502735 3553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:37.502813 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal
uuid: "0e1f5eb5e6924a76a0dfde22c63a9963"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "90aee719a18aff42fde1b4ce3851d8b4"
server_key_iv: "e20dd59e17bcdf81f9bdbfa97f5460b2"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.502864 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:37.506300 3546 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } attempt: 2
I20240501 02:10:37.506807 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.506677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46060)
0501 02:10:37.506736 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.506774 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20240501 02:10:37.507149 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.507066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46076)
0501 02:10:37.507107 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.507130 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:10:37.507236 3546 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:40773: Network error: Client connection negotiation failed: client connection to 127.22.12.125:40773: connect: Connection refused (error 111)
I20240501 02:10:37.518235 3553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:37.520386 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:37.520510 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:37.524220 3553 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.125:40773
I20240501 02:10:37.524293 3620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.125:40773 every 8 connection(s)
I20240501 02:10:37.524595 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
I20240501 02:10:37.525198 3621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:37.527022 3621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.530480 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3553
I20240501 02:10:37.530612 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-1/wal/instance
I20240501 02:10:37.530802 22577 external_mini_cluster.cc:1428] Setting key ba84cd338ba0d568d7cb9ee4127bf29e
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:37.535338 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.527686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41169 (local address 127.0.0.1:40624)
0501 02:10:37.527843 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.527875 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.528308 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.528525 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.528530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.528853 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.529139 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.529143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.529773 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.529774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.530139 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.530140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.530266 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.530885 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.530917 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.532429 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.534130 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.534137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.534140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.534343 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.534510 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.534510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.534511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.534606 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.534869 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.534873 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.534992 (+ 119us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.535082 (+ 90us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.535189 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20240501 02:10:37.535332 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.527715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.22.12.126:41169)
0501 02:10:37.527772 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.527774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.527948 (+ 174us) server_negotiation.cc:407] Connection header received
0501 02:10:37.528392 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.528392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.528401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.528464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.529228 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.529692 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.530197 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.530325 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.532544 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.532550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.532550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.532559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.534038 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.534422 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.534423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.534424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.534451 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.534665 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.534666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.534667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.534802 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.534877 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.535123 (+ 246us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.535227 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:10:37.535952 3621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.124@KRBTEST.COM" created.
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:10:37.544013 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.536230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46086)
0501 02:10:37.536273 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.536283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.536314 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.536580 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.536581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.536718 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.536858 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.536859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.537638 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.537639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.537976 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.537978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.538041 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.538464 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.538467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.541194 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.543065 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.543067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.543067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.543231 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.543384 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.543384 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.543385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.543416 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.543628 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.543629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.543688 (+ 59us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.543757 (+ 69us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.543869 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20240501 02:10:37.544031 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.536301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.22.12.125:40773)
0501 02:10:37.536465 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.536471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.536483 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:10:37.536498 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.536499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.536510 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.536565 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.536929 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.537567 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.538043 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.538232 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.541297 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.541309 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.541310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.541317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.542976 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.543302 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.543305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.543305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.543329 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.543467 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.543467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.543467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.543571 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.543633 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.543790 (+ 157us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.543935 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20240501 02:10:37.544839 3621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.545178 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.545039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57894)
0501 02:10:37.545092 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.545147 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 02:10:37.545305 3570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
I20240501 02:10:37.545727 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.545620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57898)
0501 02:10:37.545667 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.545702 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:10:37.545826 3621 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:43145: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab.
Entry for principal kudu/127.22.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab.
I20240501 02:10:37.556856 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:43145
--webserver_interface=127.22.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab
--principal=kudu/127.22.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41169,127.22.12.125:40773,127.22.12.124:43145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:37.595037 3621 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } attempt: 1
I20240501 02:10:37.595559 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.595390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57912)
0501 02:10:37.595464 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.595517 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 02:10:37.595911 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.595821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57920)
0501 02:10:37.595867 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.595893 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 02:10:37.596007 3621 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:43145: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
I20240501 02:10:37.598383 3546 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } attempt: 3
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:10:37.604717 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.598667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46094)
0501 02:10:37.598725 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.598736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.598786 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.598942 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.598943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.599090 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.599273 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.599275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.600138 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.600139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.600522 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.600523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.600579 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.601052 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.601056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.602379 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.603981 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.603982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.603983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.604115 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.604259 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.604259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.604260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.604281 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.604467 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.604468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.604505 (+ 37us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.604567 (+ 62us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.604644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240501 02:10:37.604779 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.598750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.22.12.125:40773)
0501 02:10:37.598804 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.598806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.598817 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:10:37.598845 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.598846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.598859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.598933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.599348 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.600059 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.600595 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.600779 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.602471 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.602475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.602475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.602482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.603898 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.604178 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.604179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.604180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.604204 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.604313 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.604313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.604314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.604421 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.604478 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.604603 (+ 125us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.604692 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:10:37.605142 3546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.605470 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.605378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57924)
0501 02:10:37.605422 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.605448 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240501 02:10:37.605793 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.605704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57938)
0501 02:10:37.605747 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.605769 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 02:10:37.605880 3546 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:43145: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
I20240501 02:10:37.637019 3546 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } attempt: 1
W20240501 02:10:37.636981 3628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:37.637167 3628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:37.637205 3628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 02:10:37.637499 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.637336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57954)
0501 02:10:37.637413 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.637461 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20240501 02:10:37.637842 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.637752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57956)
0501 02:10:37.637793 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.637819 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240501 02:10:37.637957 3546 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:43145: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
W20240501 02:10:37.638813 3628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:37.638854 3628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:37.638867 3628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:37.638895 3628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:37.638917 3628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:37.640559 3628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41169,127.22.12.125:40773,127.22.12.124:43145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab
--principal=kudu/127.22.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:43145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:37.640771 3628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:37.640949 3628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:37.645053 3634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:37.645483 3633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:37.645859 3636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:37.646373 3628 server_base.cc:1034] running on GCE node
I20240501 02:10:37.646656 3628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:37.646919 3628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:37.648080 3628 hybrid_clock.cc:648] HybridClock initialized: now 1714529437648057 us; error 34 us; skew 500 ppm
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:37.650476 3628 init.cc:377] Logged in from keytab as kudu/127.22.12.124@KRBTEST.COM (short username kudu)
I20240501 02:10:37.650910 3628 webserver.cc:460] Webserver started at http://127.22.12.124:38959/ using document root <none> and password file <none>
I20240501 02:10:37.651078 3628 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:37.651119 3628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:37.651235 3628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:37.652212 3628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data/instance:
uuid: "43829d0d00954956b3df8eb363a42e49"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "c6047bf8834d446805245565ca382176"
server_key_iv: "cacd2dee183bcc7dd668cfa5ad4a996a"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.652501 3628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal/instance:
uuid: "43829d0d00954956b3df8eb363a42e49"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "c6047bf8834d446805245565ca382176"
server_key_iv: "cacd2dee183bcc7dd668cfa5ad4a996a"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.653599 3628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:10:37.654191 3643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:37.654351 3628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:37.654424 3628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal
uuid: "43829d0d00954956b3df8eb363a42e49"
format_stamp: "Formatted at 2024-05-01 02:10:37 on dist-test-slave-9tkh"
server_key: "c6047bf8834d446805245565ca382176"
server_key_iv: "cacd2dee183bcc7dd668cfa5ad4a996a"
server_key_version: "encryptionkey@0"
I20240501 02:10:37.654481 3628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:37.664151 3621 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } attempt: 2
I20240501 02:10:37.664584 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.664450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57966)
0501 02:10:37.664522 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.664552 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240501 02:10:37.664964 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.664878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57972)
0501 02:10:37.664919 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.664944 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240501 02:10:37.665081 3621 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:43145: Network error: Client connection negotiation failed: client connection to 127.22.12.124:43145: connect: Connection refused (error 111)
I20240501 02:10:37.668159 3628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:37.670320 3628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:37.670420 3628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:37.674007 3628 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.124:43145
I20240501 02:10:37.674062 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.124:43145 every 8 connection(s)
I20240501 02:10:37.674345 3628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
I20240501 02:10:37.675001 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:37.676821 3696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.681185 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3628
I20240501 02:10:37.681296 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714529249419714-22577-0/minicluster-data/master-2/wal/instance
I20240501 02:10:37.681479 22577 external_mini_cluster.cc:1428] Setting key ec2e51d2a9676e422f0e7f4fe0120b5c
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:37.685309 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.677521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.22.12.126:41169)
0501 02:10:37.677610 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.677612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.677737 (+ 125us) server_negotiation.cc:407] Connection header received
0501 02:10:37.678259 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.678260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.678271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.678332 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.679172 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.679650 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.680247 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.680381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.682579 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.682583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.682584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.682591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.684088 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.684433 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.684435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.684435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.684460 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.684631 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.684632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.684632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.684743 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.684810 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.685109 (+ 299us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.685200 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 02:10:37.685355 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.677461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41169 (local address 127.0.0.1:40636)
0501 02:10:37.677625 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.677654 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.678136 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.678428 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.678432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.678783 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.679065 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.679069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.679765 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.679766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.680156 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.680158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.680271 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.680821 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.680839 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.682465 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.684164 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.684167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.684169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.684370 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.684531 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.684532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.684532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.684582 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.684808 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.684813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.684961 (+ 148us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.685086 (+ 125us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.685210 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:10:37.686455 3696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:10:37.689536 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.681880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.22.12.124:43145)
0501 02:10:37.682206 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.682212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.682227 (+ 15us) server_negotiation.cc:407] Connection header received
0501 02:10:37.682261 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.682262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.682278 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.682351 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.683236 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.684018 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.684670 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.685118 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.686846 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.686853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.686853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.686858 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.688491 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.688807 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.688810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.688811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.688850 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.689016 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.689017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.689017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.689119 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.689245 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.689344 (+ 99us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.689427 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":72,"threads_started":1}
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:10:37.692468 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.686849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.22.12.125:40773)
0501 02:10:37.686918 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.686920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.686929 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:10:37.686952 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.686952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.686962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.687013 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.687452 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.688110 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.688615 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.688812 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.690216 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.690227 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.690228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.690235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.691566 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.691867 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.691867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.691868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.691892 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.692012 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.692013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.692013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.692117 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.692184 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.692326 (+ 142us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.692396 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 02:10:37.692706 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.686764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:40773 (local address 127.0.0.1:46098)
0501 02:10:37.686815 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.686826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.686869 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.687026 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.687027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.687172 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.687372 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.687374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.688188 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.688189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.688527 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.688529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.688592 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.688990 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.688994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.690121 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.691645 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.691645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.691646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.691805 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.691946 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.691946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.691946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.691966 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.692171 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.692171 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.692213 (+ 42us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.692277 (+ 64us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.692614 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20240501 02:10:37.693179 3696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:10:37.696638 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.690318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.22.12.126:41169)
0501 02:10:37.690357 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.690359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.690431 (+ 72us) server_negotiation.cc:407] Connection header received
0501 02:10:37.690475 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.690476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.690488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.690568 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.691193 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.691622 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.692202 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.692377 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.694175 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.694179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.694179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.694187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.695701 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.696024 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.696025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.696026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.696047 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.696176 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.696177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.696177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.696288 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.696439 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.696475 (+ 36us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.696549 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:10:37.699285 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.693478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.22.12.124:43145)
0501 02:10:37.693511 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.693512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.693545 (+ 33us) server_negotiation.cc:407] Connection header received
0501 02:10:37.693606 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.693606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.693615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.693656 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.694013 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.694623 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.695225 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.695412 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.696920 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.696924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.696924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.696931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.698433 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.698736 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.698737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.698738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.698760 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.698903 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.698903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.698903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.698980 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.699023 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.699145 (+ 122us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.699215 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:10:37.699391 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.693419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:57990)
0501 02:10:37.693507 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.693516 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.693559 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.693657 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.693658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.693788 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.693946 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.693947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.694731 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.694732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.695145 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.695148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.695219 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.695556 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.695559 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.696837 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.698512 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.698513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.698513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.698666 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.698820 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.698820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.698821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.698849 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.699028 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.699028 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.699066 (+ 38us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.699132 (+ 66us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.699279 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":52}
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:10:37.701087 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: Bootstrap starting.
I20240501 02:10:37.702037 3696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:37.702328 3696 log.cc:824] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:37.703163 3696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: No bootstrap required, opened a new log
I20240501 02:10:37.703779 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.697293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.22.12.125:40773)
0501 02:10:37.697330 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.697331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.697422 (+ 91us) server_negotiation.cc:407] Connection header received
0501 02:10:37.697467 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.697469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.697478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.697532 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.698119 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.698743 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.699819 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.700068 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.701389 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.701393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.701394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.701400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.702877 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.703193 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.703194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.703194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.703214 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.703364 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.703365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.703365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.703451 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.703551 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.703635 (+ 84us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.703703 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:10:37.704695 3696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.704819 3696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:37.704841 3696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43829d0d00954956b3df8eb363a42e49, State: Initialized, Role: FOLLOWER
I20240501 02:10:37.704938 3696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.705225 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.705248 3696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:37.705361 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:10:37.705513 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20240501 02:10:37.707288 3717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:10:37.707401 3717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:10:37.707834 3628 master_runner.cc:186] Error getting master registration for 127.22.12.126:41169: OK, instance_id { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" instance_seqno: 1714529437373197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 02:10:37.710218 3628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 02:10:37.713132 3546 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } attempt: 2
I20240501 02:10:37.716898 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.706951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.22.12.124:43145)
0501 02:10:37.706991 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.706993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.707280 (+ 287us) server_negotiation.cc:407] Connection header received
0501 02:10:37.707375 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.707376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.707385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.707412 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.708295 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.708959 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.709693 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.709861 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.711328 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.711332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.711332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.711339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.713394 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.714706 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.714710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.714711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.714811 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.715340 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.715341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.715344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.715784 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.715994 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.716675 (+ 681us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.716805 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:10:37.718222 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.706952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.22.12.125:40773)
0501 02:10:37.706991 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.706993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.707487 (+ 494us) server_negotiation.cc:407] Connection header received
0501 02:10:37.707571 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.707572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.707581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.707606 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.708402 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.709041 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.709743 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.709907 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.711478 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.711482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.711482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.711488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.716432 (+ 4944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.716935 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.716936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.716936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.716961 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.717389 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.717390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.717390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.717505 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.717575 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.718036 (+ 461us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.718115 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:10:37.719463 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.707138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.22.12.126:41169)
0501 02:10:37.707178 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.707179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.707262 (+ 83us) server_negotiation.cc:407] Connection header received
0501 02:10:37.707357 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.707358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.707369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.707397 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.711161 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.711807 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.712494 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.712855 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.716813 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.716818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.716818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.716825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.718376 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.718760 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.718761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.718762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.718787 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.718937 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.718938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.718938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.719033 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.719101 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.719301 (+ 200us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.719373 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:10:37.723383 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.713807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:58000)
0501 02:10:37.713940 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.713963 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.714078 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.717268 (+ 3190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.717269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.717619 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.717824 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.717827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.718685 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.718687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.719236 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.719239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.719307 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.719741 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.719745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.721094 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.722495 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.722496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.722496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.722696 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.722857 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.722857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.722858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.722887 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.723070 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.723070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.723118 (+ 48us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.723184 (+ 66us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.723284 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20240501 02:10:37.723528 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.714887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.22.12.124:43145)
0501 02:10:37.716977 (+ 2090us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.716978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.716986 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:10:37.717005 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.717006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.717016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.717069 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.717918 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.718599 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.719471 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.719640 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.721182 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.721184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.721185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.721190 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.722405 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.722781 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.722782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.722782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.722799 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.722940 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.722940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.722940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.723017 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.723056 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.723349 (+ 293us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.723443 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2073,"thread_start_us":339,"threads_started":1}
I20240501 02:10:37.724802 3546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300: Bootstrap starting.
I20240501 02:10:37.725356 3546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:37.725590 3546 log.cc:824] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:37.726339 3546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300: No bootstrap required, opened a new log
I20240501 02:10:37.727669 3546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.727784 3546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:37.727802 3546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bed8cf7da6f402d8af87b223d1fa300, State: Initialized, Role: FOLLOWER
I20240501 02:10:37.727893 3546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.728185 3546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:37.728192 3724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.728325 3724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: This master's current role is: FOLLOWER
W20240501 02:10:37.729926 3735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:10:37.730041 3735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:10:37.730854 3478 master_runner.cc:186] Error getting master registration for 127.22.12.125:40773: OK, instance_id { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" instance_seqno: 1714529437523155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 02:10:37.733201 3478 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 02:10:37.742075 3724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:37.742203 3724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.742463 3724 leader_election.cc:288] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1f5eb5e6924a76a0dfde22c63a9963 (127.22.12.125:40773), 43829d0d00954956b3df8eb363a42e49 (127.22.12.124:43145)
I20240501 02:10:37.742836 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bed8cf7da6f402d8af87b223d1fa300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43829d0d00954956b3df8eb363a42e49" is_pre_election: true
I20240501 02:10:37.743014 3670 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bed8cf7da6f402d8af87b223d1fa300 in term 0.
I20240501 02:10:37.743227 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bed8cf7da6f402d8af87b223d1fa300, 43829d0d00954956b3df8eb363a42e49; no voters:
I20240501 02:10:37.743374 3724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:10:37.743431 3724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:10:37.743458 3724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:37.744450 3724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.744629 3724 leader_election.cc:288] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [CANDIDATE]: Term 1 election: Requested vote from peers 0e1f5eb5e6924a76a0dfde22c63a9963 (127.22.12.125:40773), 43829d0d00954956b3df8eb363a42e49 (127.22.12.124:43145)
I20240501 02:10:37.744908 3670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bed8cf7da6f402d8af87b223d1fa300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43829d0d00954956b3df8eb363a42e49"
I20240501 02:10:37.744999 3670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:37.745952 3670 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bed8cf7da6f402d8af87b223d1fa300 in term 1.
I20240501 02:10:37.746183 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bed8cf7da6f402d8af87b223d1fa300, 43829d0d00954956b3df8eb363a42e49; no voters:
I20240501 02:10:37.746361 3724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:37.746552 3724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [term 1 LEADER]: Becoming Leader. State: Replica: 3bed8cf7da6f402d8af87b223d1fa300, State: Running, Role: LEADER
I20240501 02:10:37.746649 3724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.747012 3724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bed8cf7da6f402d8af87b223d1fa300. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.747077 3724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:37.747277 3739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:37.747500 3739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:37.749372 3670 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [term 1 FOLLOWER]: Refusing update from remote peer 3bed8cf7da6f402d8af87b223d1fa300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:37.749729 3737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:37.751046 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bed8cf7da6f402d8af87b223d1fa300. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.751153 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:10:37.752259 3737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.752346 3737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:37.752626 3739 catalog_manager.cc:1347] Generated new cluster ID: 57510f11e845428bac3739388e2ef8b7
I20240501 02:10:37.752677 3739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:10:37.752995 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.753072 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:10:37.753262 3741 mvcc.cc:205] Tried to move back new op lower bound from 7022712577018408960 to 7022712577010802688. Current Snapshot: MvccSnapshot[applied={T|T < 7022712577018408960}]
I20240501 02:10:37.753530 3740 mvcc.cc:205] Tried to move back new op lower bound from 7022712577018408960 to 7022712577010802688. Current Snapshot: MvccSnapshot[applied={T|T < 7022712577018408960 or (T in {7022712577018408960})}]
I20240501 02:10:37.756232 3621 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } attempt: 3
May 01 02:10:37 dist-test-slave-9tkh krb5kdc[3465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529437, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:10:37.763406 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.756571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.22.12.124:43145)
0501 02:10:37.756629 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:37.756631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:37.756639 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:10:37.756661 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:37.756662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:37.756672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:37.756721 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:37.757206 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.757898 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.758809 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.758975 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.760762 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:37.760767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:37.760767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:37.760775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:37.762385 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.762728 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.762729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.762729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.762754 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:37.762888 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:37.762889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:37.762890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:37.762998 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:37.763055 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:37.763227 (+ 172us) server_negotiation.cc:299] Negotiation successful
0501 02:10:37.763307 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:10:37.763417 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:37.756510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:43145 (local address 127.0.0.1:58002)
0501 02:10:37.756559 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:37.756570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:37.756606 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:37.756731 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:37.756732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:37.756911 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:37.757127 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.757130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.757986 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:37.757988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:37.758517 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:37.758518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:37.758598 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:37.759134 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:37.759139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:37.760683 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:37.762469 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.762469 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.762470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.762665 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.762809 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:37.762810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:37.762810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:37.762838 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:37.763082 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:37.763083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:37.763145 (+ 62us) client_negotiation.cc:769] Sending connection context
0501 02:10:37.763212 (+ 67us) client_negotiation.cc:240] Negotiation successful
0501 02:10:37.763322 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240501 02:10:37.765085 3621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: Bootstrap starting.
I20240501 02:10:37.765743 3621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:37.765997 3621 log.cc:824] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:37.767247 3621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: No bootstrap required, opened a new log
I20240501 02:10:37.769039 3621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.769162 3621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:37.769186 3621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1f5eb5e6924a76a0dfde22c63a9963, State: Initialized, Role: FOLLOWER
I20240501 02:10:37.769296 3621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } }
I20240501 02:10:37.769594 3621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:37.770625 3593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:37.771585 3593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 1 FOLLOWER]: Refusing update from remote peer 3bed8cf7da6f402d8af87b223d1fa300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:37.771752 3595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bed8cf7da6f402d8af87b223d1fa300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" is_pre_election: true
I20240501 02:10:37.771847 3595 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bed8cf7da6f402d8af87b223d1fa300 in term 1.
I20240501 02:10:37.772070 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bed8cf7da6f402d8af87b223d1fa300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963"
I20240501 02:10:37.772814 3594 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bed8cf7da6f402d8af87b223d1fa300 in term 1.
W20240501 02:10:37.773138 3574 tablet.cc:2369] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:10:37.773617 3724 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 02:10:37.776120 3761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:10:37.776396 3761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:10:37.776297 3755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bed8cf7da6f402d8af87b223d1fa300. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.776549 3755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:10:37.776629 3755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.776723 3755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:10:37.777806 3753 mvcc.cc:205] Tried to move back new op lower bound from 7022712577018408960 to 7022712577010802688. Current Snapshot: MvccSnapshot[applied={T|T < 7022712577018408960}]
I20240501 02:10:37.778774 3746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.778918 3746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:10:37.817576 3724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bed8cf7da6f402d8af87b223d1fa300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bed8cf7da6f402d8af87b223d1fa300" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41169 } } peers { permanent_uuid: "0e1f5eb5e6924a76a0dfde22c63a9963" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 40773 } } peers { permanent_uuid: "43829d0d00954956b3df8eb363a42e49" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 43145 } } }
I20240501 02:10:37.817698 3724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:37.818341 3739 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:10:37.819141 3739 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:37.824862 3739 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3bed8cf7da6f402d8af87b223d1fa300: Generated new TSK 0
I20240501 02:10:37.825122 3739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:10:38.707912 3717 catalog_manager.cc:1259] Loaded cluster ID: 57510f11e845428bac3739388e2ef8b7
I20240501 02:10:38.707990 3717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: loading cluster ID for follower catalog manager: success
I20240501 02:10:38.708849 3717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: acquiring CA information for follower catalog manager: success
I20240501 02:10:38.709116 3717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43829d0d00954956b3df8eb363a42e49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:10:38.776836 3761 catalog_manager.cc:1259] Loaded cluster ID: 57510f11e845428bac3739388e2ef8b7
I20240501 02:10:38.776911 3761 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: loading cluster ID for follower catalog manager: success
I20240501 02:10:38.777796 3761 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: acquiring CA information for follower catalog manager: success
I20240501 02:10:38.778095 3761 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e1f5eb5e6924a76a0dfde22c63a9963: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:10:38.878829 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3478
I20240501 02:10:38.883790 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3553
I20240501 02:10:38.888249 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3628
2024-05-01T02:10:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2648 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:10:38 dist-test-slave-9tkh krb5kdc[3771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:10:38 dist-test-slave-9tkh krb5kdc[3771](info): set up 2 sockets
May 01 02:10:38 dist-test-slave-9tkh krb5kdc[3771](info): commencing operation
krb5kdc: starting...
W20240501 02:10:39.782413 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.876s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:10:39 dist-test-slave-9tkh krb5kdc[3771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:10:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:10:39.893819 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39313
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:39313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:39.972235 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:39.972442 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:39.972473 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:39.973838 3784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:39.973893 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:39.973912 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:39.973927 3784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:39.973942 3784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:39.975688 3784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:39313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:39.975937 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:39.976127 3784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:39.980521 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:39.980942 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:39.981390 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:39.981877 3784 server_base.cc:1034] running on GCE node
I20240501 02:10:39.982113 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:39.982370 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:39.983494 3784 hybrid_clock.cc:648] HybridClock initialized: now 1714529439983478 us; error 40 us; skew 500 ppm
May 01 02:10:39 dist-test-slave-9tkh krb5kdc[3771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529439, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:39.985972 3784 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:39.986395 3784 webserver.cc:460] Webserver started at http://127.22.12.126:39755/ using document root <none> and password file <none>
I20240501 02:10:39.986594 3784 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:39.986635 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:39.986747 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:39.987756 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "956b9078a1e84374a8cdc68b3732af84"
format_stamp: "Formatted at 2024-05-01 02:10:39 on dist-test-slave-9tkh"
server_key: "2f7cac45e2c3293de83277d2b2ac5206"
server_key_iv: "dc696d3821cfbca0caebde1be5a09d2d"
server_key_version: "encryptionkey@0"
I20240501 02:10:39.988042 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "956b9078a1e84374a8cdc68b3732af84"
format_stamp: "Formatted at 2024-05-01 02:10:39 on dist-test-slave-9tkh"
server_key: "2f7cac45e2c3293de83277d2b2ac5206"
server_key_iv: "dc696d3821cfbca0caebde1be5a09d2d"
server_key_version: "encryptionkey@0"
I20240501 02:10:39.989192 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:10:39.989883 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:39.990065 3784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:39.990145 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "956b9078a1e84374a8cdc68b3732af84"
format_stamp: "Formatted at 2024-05-01 02:10:39 on dist-test-slave-9tkh"
server_key: "2f7cac45e2c3293de83277d2b2ac5206"
server_key_iv: "dc696d3821cfbca0caebde1be5a09d2d"
server_key_version: "encryptionkey@0"
I20240501 02:10:39.990206 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:39.998661 3784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:40.000672 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:40.000789 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:40.004518 3784 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:39313
I20240501 02:10:40.004563 3851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:39313 every 8 connection(s)
I20240501 02:10:40.004874 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:40.005704 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:40.007896 3852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84: Bootstrap starting.
I20240501 02:10:40.008174 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3784
I20240501 02:10:40.008262 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:10:40.008409 22577 external_mini_cluster.cc:1428] Setting key 0556866fc8e90317c2185df89886782c
I20240501 02:10:40.008489 3852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:40.008762 3852 log.cc:824] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:40.009482 3852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84: No bootstrap required, opened a new log
I20240501 02:10:40.010991 3852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956b9078a1e84374a8cdc68b3732af84" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39313 } }
I20240501 02:10:40.011165 3852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:40.011190 3852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956b9078a1e84374a8cdc68b3732af84, State: Initialized, Role: FOLLOWER
I20240501 02:10:40.011314 3852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956b9078a1e84374a8cdc68b3732af84" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39313 } }
I20240501 02:10:40.011394 3852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:40.011427 3852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:40.011471 3852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:40.012125 3852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956b9078a1e84374a8cdc68b3732af84" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39313 } }
I20240501 02:10:40.012230 3852 leader_election.cc:302] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 956b9078a1e84374a8cdc68b3732af84; no voters:
I20240501 02:10:40.012404 3852 leader_election.cc:288] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:10:40.012457 3857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:40.012645 3857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [term 1 LEADER]: Becoming Leader. State: Replica: 956b9078a1e84374a8cdc68b3732af84, State: Running, Role: LEADER
I20240501 02:10:40.012655 3852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:40.012810 3857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956b9078a1e84374a8cdc68b3732af84" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39313 } }
I20240501 02:10:40.013180 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "956b9078a1e84374a8cdc68b3732af84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956b9078a1e84374a8cdc68b3732af84" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39313 } } }
I20240501 02:10:40.013249 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:40.013442 3859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 956b9078a1e84374a8cdc68b3732af84. Latest consensus state: current_term: 1 leader_uuid: "956b9078a1e84374a8cdc68b3732af84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956b9078a1e84374a8cdc68b3732af84" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39313 } } }
I20240501 02:10:40.013547 3859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84 [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:40.013679 3863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:40.014192 3863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 01 02:10:40 dist-test-slave-9tkh krb5kdc[3771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:40.015792 3863 catalog_manager.cc:1347] Generated new cluster ID: 8c11362a230d4c9690624682e7a7f55a
I20240501 02:10:40.015841 3863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:10:40.019479 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:40.008878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.22.12.126:39313)
0501 02:10:40.009103 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:40.009113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:40.009129 (+ 16us) server_negotiation.cc:407] Connection header received
0501 02:10:40.009726 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:40.009735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:40.009803 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:40.009956 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:40.010724 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:40.012231 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:40.012961 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:40.013200 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:40.015783 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:40.015807 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:40.015807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:40.015826 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:40.017852 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:40.018232 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:40.018236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:40.018236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:40.018313 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:40.018480 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:40.018481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:40.018481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:40.018760 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:40.018867 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:40.019135 (+ 268us) server_negotiation.cc:299] Negotiation successful
0501 02:10:40.019284 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20240501 02:10:40.020615 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:10:40.021494 3863 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:10:40.022024 3863 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:40.026396 3863 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 956b9078a1e84374a8cdc68b3732af84: Generated new TSK 0
I20240501 02:10:40.026612 3863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:10:40.074357 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3784
2024-05-01T02:10:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1185 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:10:40 dist-test-slave-9tkh krb5kdc[3879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:10:40 dist-test-slave-9tkh krb5kdc[3879](info): set up 2 sockets
May 01 02:10:40 dist-test-slave-9tkh krb5kdc[3879](info): commencing operation
krb5kdc: starting...
W20240501 02:10:40.958400 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.866s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:10:41.069363 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45189
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:45189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:41.147238 3892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:41.147431 3892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:41.147461 3892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:41.148785 3892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:41.148831 3892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:41.148849 3892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:41.148865 3892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:41.148880 3892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:41.150367 3892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:45189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:41.150640 3892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:41.150816 3892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:41.155211 3897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.155633 3900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.156035 3898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:41.156483 3892 server_base.cc:1034] running on GCE node
I20240501 02:10:41.156680 3892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:41.156921 3892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:41.158046 3892 hybrid_clock.cc:648] HybridClock initialized: now 1714529441158027 us; error 36 us; skew 500 ppm
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:41.160368 3892 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:41.160791 3892 webserver.cc:460] Webserver started at http://127.22.12.126:33367/ using document root <none> and password file <none>
I20240501 02:10:41.160965 3892 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:41.161007 3892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:41.161108 3892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:41.162098 3892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "79fdf156c108453184292c3e12f9ca0c"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "d53464c5ba584eaaeac7b3bde102c0b1"
server_key_iv: "eb82ec461ebe1b6c3e6ad1fd35c75db1"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.162401 3892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "79fdf156c108453184292c3e12f9ca0c"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "d53464c5ba584eaaeac7b3bde102c0b1"
server_key_iv: "eb82ec461ebe1b6c3e6ad1fd35c75db1"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.163625 3892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:10:41.164389 3907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.164567 3892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:10:41.164644 3892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "79fdf156c108453184292c3e12f9ca0c"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "d53464c5ba584eaaeac7b3bde102c0b1"
server_key_iv: "eb82ec461ebe1b6c3e6ad1fd35c75db1"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.164705 3892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:41.178058 3892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:41.180290 3892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:41.180423 3892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:41.184216 3892 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:45189
I20240501 02:10:41.184273 3959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:45189 every 8 connection(s)
I20240501 02:10:41.184612 3892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:41.185515 3960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:41.187786 3960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap starting.
I20240501 02:10:41.188344 3960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:41.188603 3960 log.cc:824] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:41.189457 3960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: No bootstrap required, opened a new log
I20240501 02:10:41.191597 3960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:41.191761 3960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:41.191797 3960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fdf156c108453184292c3e12f9ca0c, State: Initialized, Role: FOLLOWER
I20240501 02:10:41.191931 3960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:41.192011 3960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:41.192040 3960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:41.192085 3960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:41.192842 3960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:41.192952 3960 leader_election.cc:302] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79fdf156c108453184292c3e12f9ca0c; no voters:
I20240501 02:10:41.193128 3960 leader_election.cc:288] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:10:41.193406 3960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:41.193439 3963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:41.193610 3963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 LEADER]: Becoming Leader. State: Replica: 79fdf156c108453184292c3e12f9ca0c, State: Running, Role: LEADER
I20240501 02:10:41.193858 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3892
I20240501 02:10:41.193974 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:10:41.194147 22577 external_mini_cluster.cc:1428] Setting key ff1e4eef90726480c0ed9997cb28ea9b
I20240501 02:10:41.194840 3963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:41.195461 3964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79fdf156c108453184292c3e12f9ca0c. Latest consensus state: current_term: 1 leader_uuid: "79fdf156c108453184292c3e12f9ca0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } } }
I20240501 02:10:41.195564 3964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:41.195958 3965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79fdf156c108453184292c3e12f9ca0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } } }
W20240501 02:10:41.196183 3979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240501 02:10:41.196477 3965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: This master's current role is: LEADER
W20240501 02:10:41.196485 3979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:10:41.197803 3981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:41.198017 3981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:41.199702 3981 catalog_manager.cc:1347] Generated new cluster ID: 398ca7e61f204fe6ac0c4459d95cd17e
I20240501 02:10:41.199761 3981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:41.203766 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.194713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.22.12.126:45189)
0501 02:10:41.194844 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.194856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.194907 (+ 51us) server_negotiation.cc:407] Connection header received
0501 02:10:41.195078 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.195092 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.195136 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.196335 (+ 1199us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:41.196470 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.197621 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.198243 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.198488 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.200636 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:41.200658 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:41.200658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:41.200676 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:41.202258 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.202591 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.202594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.202594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.202645 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.202807 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.202808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.202808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.203002 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:41.203131 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.203279 (+ 148us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.203408 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":78,"threads_started":1}
I20240501 02:10:41.209141 3981 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:10:41.209877 3981 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:41.218847 3981 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Generated new TSK 0
I20240501 02:10:41.219067 3981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:10:41.254464 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45189
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:41.357357 3986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:41.357561 3986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:41.357594 3986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:41.358971 3986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:41.359021 3986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:41.359055 3986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:10:41.360695 3986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:41.360920 3986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:41.361124 3986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:41.365972 3992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.366350 3994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.366876 3991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:41.367223 3986 server_base.cc:1034] running on GCE node
I20240501 02:10:41.367401 3986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:41.367636 3986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:41.368773 3986 hybrid_clock.cc:648] HybridClock initialized: now 1714529441368759 us; error 32 us; skew 500 ppm
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:41.371371 3986 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:10:41.371812 3986 webserver.cc:460] Webserver started at http://127.22.12.65:45331/ using document root <none> and password file <none>
I20240501 02:10:41.371984 3986 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:41.372023 3986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:41.372125 3986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:41.373155 3986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "7f3bb53c321540d8a2310ec9a5d05f17"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "87d5fc7a5bfb4d60581e94f7dbe1ddcb"
server_key_iv: "f94d3916ba3a42ad5a0bc488696de1f6"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.373457 3986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "7f3bb53c321540d8a2310ec9a5d05f17"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "87d5fc7a5bfb4d60581e94f7dbe1ddcb"
server_key_iv: "f94d3916ba3a42ad5a0bc488696de1f6"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.374610 3986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:10:41.375257 4001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.375433 3986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:41.375530 3986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "7f3bb53c321540d8a2310ec9a5d05f17"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "87d5fc7a5bfb4d60581e94f7dbe1ddcb"
server_key_iv: "f94d3916ba3a42ad5a0bc488696de1f6"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.375597 3986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:41.393116 3986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:41.395593 3986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:41.395697 3986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:41.395889 3986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:41.396203 3986 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:41.396236 3986 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.396265 3986 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:41.396281 3986 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.401602 3986 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:37069
I20240501 02:10:41.401664 4114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:37069 every 8 connection(s)
I20240501 02:10:41.401945 3986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:41.409834 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.402671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:43133 (local address 127.22.12.126:45189)
0501 02:10:41.402731 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.402734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.402931 (+ 197us) server_negotiation.cc:407] Connection header received
0501 02:10:41.403527 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.403529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.403543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.403625 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:41.404683 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.405170 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.405657 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.405764 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.407643 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:41.407649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:41.407649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:41.407655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:41.408873 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.409194 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.409195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.409195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.409212 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.409341 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.409341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.409341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.409431 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:41.409471 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.409680 (+ 209us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.409756 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:10:41.410041 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.402666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:43133)
0501 02:10:41.402815 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.402841 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.403356 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.403702 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.403706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.404013 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:41.404597 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.404602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.405234 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.405234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.405598 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.405599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.405698 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.406232 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:41.406248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:41.407554 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:41.408944 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:41.408948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:41.408951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:41.409137 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:41.409255 (+ 118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:41.409255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:41.409256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:41.409303 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:41.409470 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:41.409473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:41.409580 (+ 107us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.409664 (+ 84us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.409781 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20240501 02:10:41.410056 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 3986
I20240501 02:10:41.410157 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:10:41.410374 22577 external_mini_cluster.cc:1428] Setting key adffd65071d1674a7234beddf1cbf7e1
I20240501 02:10:41.410624 4115 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45189
I20240501 02:10:41.410720 4115 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:41.410928 4115 heartbeater.cc:511] Master 127.22.12.126:45189 requested a full tablet report, sending...
I20240501 02:10:41.411396 3924 ts_manager.cc:194] Registered new tserver with Master: 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069)
I20240501 02:10:41.412017 3924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:43133
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:10:41.433717 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45189
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:41.516203 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:41.516428 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:41.516453 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:41.518016 4120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:41.518064 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:41.518090 4120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:10:41.519872 4120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:41.520093 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:41.520282 4120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:41.524633 4126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.525099 4125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.525517 4128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:41.526013 4120 server_base.cc:1034] running on GCE node
I20240501 02:10:41.526180 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:41.526402 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:41.527542 4120 hybrid_clock.cc:648] HybridClock initialized: now 1714529441527527 us; error 32 us; skew 500 ppm
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:41.530141 4120 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:10:41.530615 4120 webserver.cc:460] Webserver started at http://127.22.12.66:33049/ using document root <none> and password file <none>
I20240501 02:10:41.530793 4120 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:41.530835 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:41.530939 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:41.531939 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "88a97f00646947d7ab43045491d3c2d0"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "bc9f999eeb28d8ce9a7794f0386943a4"
server_key_iv: "964324158960a942f589b3569a308964"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.532230 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "88a97f00646947d7ab43045491d3c2d0"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "bc9f999eeb28d8ce9a7794f0386943a4"
server_key_iv: "964324158960a942f589b3569a308964"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.533370 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:10:41.533990 4135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.534153 4120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:10:41.534231 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "88a97f00646947d7ab43045491d3c2d0"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "bc9f999eeb28d8ce9a7794f0386943a4"
server_key_iv: "964324158960a942f589b3569a308964"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.534289 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:41.560015 4120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:41.561808 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:41.561923 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:41.562145 4120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:41.562448 4120 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:41.562480 4120 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.562518 4120 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:41.562572 4120 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.568022 4120 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:43529
I20240501 02:10:41.568070 4248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:43529 every 8 connection(s)
I20240501 02:10:41.568398 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:41.576452 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.569075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:47403 (local address 127.22.12.126:45189)
0501 02:10:41.569129 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.569131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.569313 (+ 182us) server_negotiation.cc:407] Connection header received
0501 02:10:41.569761 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.569762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.569773 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.569828 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:41.570867 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.571342 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.571832 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.571931 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.574068 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:41.574074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:41.574075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:41.574082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:41.575452 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.575772 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.575773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.575773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.575792 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.575929 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.575930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.575930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.576023 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:41.576065 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.576304 (+ 239us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.576382 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:10:41.576516 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.569051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:47403)
0501 02:10:41.569196 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.569222 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.569668 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.569887 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.569890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.570182 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:41.570793 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.570798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.571409 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.571409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.571778 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.571779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.571880 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.572389 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:41.572406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:41.573908 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:41.575526 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:41.575530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:41.575532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:41.575716 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:41.575838 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:41.575838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:41.575839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:41.575887 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:41.576062 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:41.576065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:41.576172 (+ 107us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.576254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.576359 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20240501 02:10:41.577031 4249 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45189
I20240501 02:10:41.577126 4249 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:41.577330 4249 heartbeater.cc:511] Master 127.22.12.126:45189 requested a full tablet report, sending...
I20240501 02:10:41.577773 3924 ts_manager.cc:194] Registered new tserver with Master: 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529)
I20240501 02:10:41.578145 3924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:47403
I20240501 02:10:41.578563 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4120
I20240501 02:10:41.578675 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:10:41.578874 22577 external_mini_cluster.cc:1428] Setting key 96b5b3b4c102f2e4b05dbeda1243698e
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:10:41.603667 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45189
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:41.684285 4254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:41.684500 4254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:41.684525 4254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:41.685899 4254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:41.685945 4254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:41.685971 4254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:10:41.687752 4254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:41.687959 4254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:41.688135 4254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:41.692843 4260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.693331 4262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.693799 4259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:41.694375 4254 server_base.cc:1034] running on GCE node
I20240501 02:10:41.694666 4254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:41.694957 4254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:41.696095 4254 hybrid_clock.cc:648] HybridClock initialized: now 1714529441696077 us; error 34 us; skew 500 ppm
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:41.698616 4254 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:10:41.699095 4254 webserver.cc:460] Webserver started at http://127.22.12.67:37957/ using document root <none> and password file <none>
I20240501 02:10:41.699287 4254 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:41.699329 4254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:41.699440 4254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:41.700456 4254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "2cb90544166949c99087cec1f4f17b0a"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "1fb03b1bf4bc98947bf06c9b8df60501"
server_key_iv: "ddafa4371135f497be630823337752ff"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.700769 4254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "2cb90544166949c99087cec1f4f17b0a"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "1fb03b1bf4bc98947bf06c9b8df60501"
server_key_iv: "ddafa4371135f497be630823337752ff"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.701954 4254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:10:41.702704 4269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.702857 4254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:41.703002 4254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "2cb90544166949c99087cec1f4f17b0a"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "1fb03b1bf4bc98947bf06c9b8df60501"
server_key_iv: "ddafa4371135f497be630823337752ff"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.703088 4254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:41.715301 4254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:41.717341 4254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:41.717470 4254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:41.717694 4254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:41.718382 4254 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:41.718461 4254 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.718590 4254 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:41.718653 4254 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.724143 4254 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:41307
I20240501 02:10:41.724182 4382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:41307 every 8 connection(s)
I20240501 02:10:41.724493 4254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:10:41.728395 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4254
I20240501 02:10:41.728494 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:10:41.728677 22577 external_mini_cluster.cc:1428] Setting key 359a1131de96b2be51da46b1a7dc2f2b
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:41.732970 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.725247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:34479 (local address 127.22.12.126:45189)
0501 02:10:41.725317 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.725319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.725463 (+ 144us) server_negotiation.cc:407] Connection header received
0501 02:10:41.725969 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.725971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.725982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.726037 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:41.727151 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.727650 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.728207 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.728377 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.730598 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:41.730602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:41.730603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:41.730610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:41.731792 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.732161 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.732163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.732163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.732187 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.732360 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.732361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.732361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.732466 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:41.732521 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.732779 (+ 258us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.732876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:10:41.733074 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.725202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:34479)
0501 02:10:41.725349 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.725381 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.725861 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.726105 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.726109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.726423 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:41.727045 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.727050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.727741 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.727742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.728121 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.728123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.728257 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.728916 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:41.728941 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:41.730482 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:41.731878 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:41.731884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:41.731887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:41.732092 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:41.732243 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:41.732244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:41.732244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:41.732306 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:41.732523 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:41.732529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:41.732652 (+ 123us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.732748 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.732887 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20240501 02:10:41.733532 4383 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45189
I20240501 02:10:41.733623 4383 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:41.733803 4383 heartbeater.cc:511] Master 127.22.12.126:45189 requested a full tablet report, sending...
I20240501 02:10:41.734159 3924 ts_manager.cc:194] Registered new tserver with Master: 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
I20240501 02:10:41.734508 3924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:34479
I20240501 02:10:41.740496 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:10:41.746968 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.741790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.22.12.126:45189)
0501 02:10:41.741851 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.741852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.741907 (+ 55us) server_negotiation.cc:407] Connection header received
0501 02:10:41.741953 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.741953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.741961 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.742009 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:41.742689 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.743174 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.743680 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.743804 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.744743 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:41.744746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:41.744746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:41.744750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:41.746037 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.746330 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.746331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.746331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.746348 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:41.746460 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:41.746460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:41.746460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:41.746645 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:41.746724 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.746808 (+ 84us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.746886 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:10:41.748611 3924 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:10:41.750056 3924 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:10:41.755730 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.752298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43529 (local address 127.0.0.1:50828)
0501 02:10:41.752468 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.752497 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.752568 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.753244 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.753245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.753247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:41.753548 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.753550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.754801 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.754801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.755515 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.755517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.755575 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.755579 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.755605 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.755644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":53,"threads_started":1}
I20240501 02:10:41.756062 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.752309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37069 (local address 127.0.0.1:51778)
0501 02:10:41.752640 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.752649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.752714 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.752921 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.752924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.752926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:41.753210 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.753212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.754242 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.754244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.755078 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.755079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.755902 (+ 823us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.755907 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.755959 (+ 52us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.756000 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":90,"threads_started":1}
I20240501 02:10:41.755656 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.752507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41307 (local address 127.0.0.1:38532)
0501 02:10:41.752702 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.752710 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.752751 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.752924 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.752925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.752926 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:41.753210 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.753212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.754549 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.754551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.755357 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.755359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.755417 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.755460 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.755515 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.755559 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":61,"threads_started":1}
I20240501 02:10:41.756383 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.752411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.22.12.65:37069)
0501 02:10:41.752589 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.752597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.752731 (+ 134us) server_negotiation.cc:407] Connection header received
0501 02:10:41.752798 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.752800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.752819 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.752904 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:41.753344 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.754164 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.755266 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.755820 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.755873 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.756257 (+ 384us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.756309 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20240501 02:10:41.756412 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.752602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.22.12.67:41307)
0501 02:10:41.752782 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.752790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.752798 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:10:41.752831 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.752832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.752849 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.752923 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:41.753297 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.754442 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.755716 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.756225 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.756275 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.756304 (+ 29us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.756346 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20240501 02:10:41.756565 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.752407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.22.12.66:43529)
0501 02:10:41.753042 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.753048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.753055 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:10:41.753083 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.753084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.753096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.753160 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:41.753647 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.754719 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.755609 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.756358 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.756408 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.756442 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.756490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":66,"threads_started":1}
I20240501 02:10:41.756922 4317 tablet_service.cc:1467] Processing CreateTablet for tablet 443dc1afa3de4122a758f66fdd84d829 (DEFAULT_TABLE table=test-table [id=116b202c7a304a1cbb51ff160dd1702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:41.757040 4183 tablet_service.cc:1467] Processing CreateTablet for tablet 443dc1afa3de4122a758f66fdd84d829 (DEFAULT_TABLE table=test-table [id=116b202c7a304a1cbb51ff160dd1702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:41.757208 4317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443dc1afa3de4122a758f66fdd84d829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:41.757299 4183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443dc1afa3de4122a758f66fdd84d829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:41.758919 4049 tablet_service.cc:1467] Processing CreateTablet for tablet 443dc1afa3de4122a758f66fdd84d829 (DEFAULT_TABLE table=test-table [id=116b202c7a304a1cbb51ff160dd1702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:41.759191 4049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443dc1afa3de4122a758f66fdd84d829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:41.759827 4400 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Bootstrap starting.
I20240501 02:10:41.760133 4401 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Bootstrap starting.
I20240501 02:10:41.760531 4400 tablet_bootstrap.cc:654] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:41.760702 4401 tablet_bootstrap.cc:654] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:41.760931 4400 log.cc:824] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:41.760974 4401 log.cc:824] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:41.761651 4400 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: No bootstrap required, opened a new log
I20240501 02:10:41.761716 4400 ts_tablet_manager.cc:1367] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:10:41.761919 4401 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: No bootstrap required, opened a new log
I20240501 02:10:41.761986 4401 ts_tablet_manager.cc:1367] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:10:41.762688 4404 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap starting.
I20240501 02:10:41.763130 4404 tablet_bootstrap.cc:654] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:41.763372 4404 log.cc:824] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:41.763847 4400 raft_consensus.cc:348] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.763993 4400 raft_consensus.cc:374] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:41.763986 4404 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: No bootstrap required, opened a new log
I20240501 02:10:41.764016 4400 raft_consensus.cc:729] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb90544166949c99087cec1f4f17b0a, State: Initialized, Role: FOLLOWER
I20240501 02:10:41.764045 4404 ts_tablet_manager.cc:1367] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:41.764086 4401 raft_consensus.cc:348] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.764143 4400 consensus_queue.cc:260] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.764206 4401 raft_consensus.cc:374] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:41.764235 4401 raft_consensus.cc:729] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a97f00646947d7ab43045491d3c2d0, State: Initialized, Role: FOLLOWER
I20240501 02:10:41.764372 4400 ts_tablet_manager.cc:1398] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:41.764357 4401 consensus_queue.cc:260] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.764484 4383 heartbeater.cc:503] Master 127.22.12.126:45189 was elected leader, sending a full tablet report...
I20240501 02:10:41.764981 4401 ts_tablet_manager.cc:1398] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:41.765090 4249 heartbeater.cc:503] Master 127.22.12.126:45189 was elected leader, sending a full tablet report...
I20240501 02:10:41.765969 4404 raft_consensus.cc:348] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.766090 4404 raft_consensus.cc:374] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:41.766112 4404 raft_consensus.cc:729] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3bb53c321540d8a2310ec9a5d05f17, State: Initialized, Role: FOLLOWER
I20240501 02:10:41.766192 4404 consensus_queue.cc:260] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.766392 4404 ts_tablet_manager.cc:1398] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 02:10:41.766441 4115 heartbeater.cc:503] Master 127.22.12.126:45189 was elected leader, sending a full tablet report...
W20240501 02:10:41.818944 4250 tablet.cc:2369] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:10:41.829648 4406 raft_consensus.cc:482] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:41.829777 4406 raft_consensus.cc:504] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.830148 4406 leader_election.cc:288] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069)
I20240501 02:10:41.832252 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.830356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37069 (local address 127.22.12.67:47621)
0501 02:10:41.830414 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.830426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.830470 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.830627 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.830628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.830630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:41.830808 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.830809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.831468 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.831469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.832038 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.832040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.832106 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.832110 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.832137 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.832174 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240501 02:10:41.832509 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.830359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43529 (local address 127.22.12.67:49467)
0501 02:10:41.830601 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.830611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:41.830686 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:41.830819 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:41.830820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:41.830822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:41.831021 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.831022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.831691 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.831692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:41.832281 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:41.832283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.832355 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.832360 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:10:41.832386 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:10:41.832422 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":17,"thread_start_us":131,"threads_started":1}
I20240501 02:10:41.832698 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.830437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:47621 (local address 127.22.12.65:37069)
0501 02:10:41.830481 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.830482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.830494 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:10:41.830544 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.830545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.830555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.830604 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:41.830891 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.831376 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.832136 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.832590 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.832600 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.832623 (+ 23us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.832647 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:10:41.832984 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.830429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:49467 (local address 127.22.12.66:43529)
0501 02:10:41.830479 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.830482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.830690 (+ 208us) server_negotiation.cc:407] Connection header received
0501 02:10:41.830742 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.830743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.830752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.830800 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:41.831104 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.831602 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.832467 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.832861 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.832868 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.832890 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.832921 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:10:41.833056 4069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443dc1afa3de4122a758f66fdd84d829" candidate_uuid: "2cb90544166949c99087cec1f4f17b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" is_pre_election: true
I20240501 02:10:41.833222 4069 raft_consensus.cc:2454] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb90544166949c99087cec1f4f17b0a in term 0.
I20240501 02:10:41.833271 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443dc1afa3de4122a758f66fdd84d829" candidate_uuid: "2cb90544166949c99087cec1f4f17b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a97f00646947d7ab43045491d3c2d0" is_pre_election: true
I20240501 02:10:41.833431 4203 raft_consensus.cc:2454] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb90544166949c99087cec1f4f17b0a in term 0.
I20240501 02:10:41.833472 4273 leader_election.cc:302] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb90544166949c99087cec1f4f17b0a, 7f3bb53c321540d8a2310ec9a5d05f17; no voters:
I20240501 02:10:41.833621 4406 raft_consensus.cc:2781] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:10:41.833706 4406 raft_consensus.cc:482] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:10:41.833729 4406 raft_consensus.cc:3037] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:41.834424 4406 raft_consensus.cc:504] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.834592 4406 leader_election.cc:288] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [CANDIDATE]: Term 1 election: Requested vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069)
I20240501 02:10:41.834791 4069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443dc1afa3de4122a758f66fdd84d829" candidate_uuid: "2cb90544166949c99087cec1f4f17b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f3bb53c321540d8a2310ec9a5d05f17"
I20240501 02:10:41.834794 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443dc1afa3de4122a758f66fdd84d829" candidate_uuid: "2cb90544166949c99087cec1f4f17b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a97f00646947d7ab43045491d3c2d0"
I20240501 02:10:41.834864 4203 raft_consensus.cc:3037] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:41.834865 4069 raft_consensus.cc:3037] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:41.835563 4203 raft_consensus.cc:2454] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb90544166949c99087cec1f4f17b0a in term 1.
I20240501 02:10:41.835558 4069 raft_consensus.cc:2454] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb90544166949c99087cec1f4f17b0a in term 1.
I20240501 02:10:41.835752 4271 leader_election.cc:302] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb90544166949c99087cec1f4f17b0a, 88a97f00646947d7ab43045491d3c2d0; no voters:
I20240501 02:10:41.835868 4406 raft_consensus.cc:2781] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:41.836154 4406 raft_consensus.cc:686] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 1 LEADER]: Becoming Leader. State: Replica: 2cb90544166949c99087cec1f4f17b0a, State: Running, Role: LEADER
I20240501 02:10:41.836268 4406 consensus_queue.cc:237] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:41.837210 3923 catalog_manager.cc:5484] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb90544166949c99087cec1f4f17b0a (127.22.12.67). New cstate: current_term: 1 leader_uuid: "2cb90544166949c99087cec1f4f17b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } health_report { overall_health: UNKNOWN } } }
I20240501 02:10:41.850066 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.847835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.22.12.67:41307)
0501 02:10:41.847890 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:41.847892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:41.847899 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:10:41.847941 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:41.847941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:41.847958 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:41.848029 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:10:41.848330 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.848845 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:41.849471 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:41.849661 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:41.849706 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:41.849794 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:41.849864 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:41.849975 (+ 111us) server_negotiation.cc:299] Negotiation successful
0501 02:10:41.850003 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:10:41.851801 4069 raft_consensus.cc:1261] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Refusing update from remote peer 2cb90544166949c99087cec1f4f17b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:41.851802 4203 raft_consensus.cc:1261] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Refusing update from remote peer 2cb90544166949c99087cec1f4f17b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:41.852145 4406 consensus_queue.cc:1035] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:41.852286 4410 consensus_queue.cc:1035] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:41.854424 4413 mvcc.cc:205] Tried to move back new op lower bound from 7022712593822547968 to 7022712593762152448. Current Snapshot: MvccSnapshot[applied={T|T < 7022712593822547968}]
I20240501 02:10:41.855113 4415 mvcc.cc:205] Tried to move back new op lower bound from 7022712593822547968 to 7022712593762152448. Current Snapshot: MvccSnapshot[applied={T|T < 7022712593822547968 or (T in {7022712593822547968})}]
I20240501 02:10:41.856873 3923 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:41.856969 3923 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:41.857708 3923 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Sending DeleteTablet for 3 replicas of tablet 443dc1afa3de4122a758f66fdd84d829
I20240501 02:10:41.858045 4317 tablet_service.cc:1514] Processing DeleteTablet for tablet 443dc1afa3de4122a758f66fdd84d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:41 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:38532
I20240501 02:10:41.858038 4049 tablet_service.cc:1514] Processing DeleteTablet for tablet 443dc1afa3de4122a758f66fdd84d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:41 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:51778
I20240501 02:10:41.858083 4183 tablet_service.cc:1514] Processing DeleteTablet for tablet 443dc1afa3de4122a758f66fdd84d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:41 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50828
I20240501 02:10:41.858307 4423 tablet_replica.cc:330] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: stopping tablet replica
I20240501 02:10:41.858364 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3986
I20240501 02:10:41.858412 4423 raft_consensus.cc:2229] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:41.859061 4424 tablet_replica.cc:330] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: stopping tablet replica
I20240501 02:10:41.859134 4424 raft_consensus.cc:2229] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:10:41.859215 4424 raft_consensus.cc:2258] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:41.859364 4425 tablet_replica.cc:330] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: stopping tablet replica
I20240501 02:10:41.859438 4425 raft_consensus.cc:2229] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:41.859521 4425 raft_consensus.cc:2258] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:41.859671 4424 ts_tablet_manager.cc:1875] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:41.859891 4425 ts_tablet_manager.cc:1875] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:41.861339 4425 ts_tablet_manager.cc:1888] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:41.861349 4424 ts_tablet_manager.cc:1888] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:41.861402 4424 log.cc:1196] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/443dc1afa3de4122a758f66fdd84d829
I20240501 02:10:41.861402 4425 log.cc:1196] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/443dc1afa3de4122a758f66fdd84d829
I20240501 02:10:41.861656 4424 ts_tablet_manager.cc:1909] T 443dc1afa3de4122a758f66fdd84d829 P 2cb90544166949c99087cec1f4f17b0a: Deleting consensus metadata
I20240501 02:10:41.861658 4425 ts_tablet_manager.cc:1909] T 443dc1afa3de4122a758f66fdd84d829 P 88a97f00646947d7ab43045491d3c2d0: Deleting consensus metadata
I20240501 02:10:41.862103 3909 catalog_manager.cc:4830] TS 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): tablet 443dc1afa3de4122a758f66fdd84d829 (table test-table [id=116b202c7a304a1cbb51ff160dd1702c]) successfully deleted
I20240501 02:10:41.862185 3909 catalog_manager.cc:4830] TS 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): tablet 443dc1afa3de4122a758f66fdd84d829 (table test-table [id=116b202c7a304a1cbb51ff160dd1702c]) successfully deleted
W20240501 02:10:41.864691 3911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:37069 (error 108)
I20240501 02:10:41.864941 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 4120
I20240501 02:10:41.865250 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:41.865136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37069 (local address 127.0.0.1:51780)
0501 02:10:41.865184 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:41.865215 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:10:41.865634 3911 catalog_manager.cc:4573] TS 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 443dc1afa3de4122a758f66fdd84d829: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37069: connect: Connection refused (error 111)
I20240501 02:10:41.870411 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 4254
I20240501 02:10:41.875435 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 3892
I20240501 02:10:41.880625 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45189
--webserver_interface=127.22.12.126
--webserver_port=33367
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:45189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:41.959600 4427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:41.959810 4427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:41.959841 4427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:41.961139 4427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:41.961184 4427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:41.961201 4427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:41.961217 4427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:41.961231 4427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:41.962790 4427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:45189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:41.963012 4427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:41.963192 4427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:41.970933 4433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.971232 4436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:41.971722 4434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:41.972167 4427 server_base.cc:1034] running on GCE node
I20240501 02:10:41.972411 4427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:41.972668 4427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:41.973809 4427 hybrid_clock.cc:648] HybridClock initialized: now 1714529441973792 us; error 37 us; skew 500 ppm
May 01 02:10:41 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529441, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:41.976214 4427 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:41.976629 4427 webserver.cc:460] Webserver started at http://127.22.12.126:33367/ using document root <none> and password file <none>
I20240501 02:10:41.976825 4427 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:41.976876 4427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:41.978186 4427 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:41.978840 4443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:41.979032 4427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:41.979107 4427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "79fdf156c108453184292c3e12f9ca0c"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "d53464c5ba584eaaeac7b3bde102c0b1"
server_key_iv: "eb82ec461ebe1b6c3e6ad1fd35c75db1"
server_key_version: "encryptionkey@0"
I20240501 02:10:41.979461 4427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:41.988684 4427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:41.990614 4427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:41.990725 4427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:41.994633 4427 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:45189
I20240501 02:10:41.994704 4495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:45189 every 8 connection(s)
I20240501 02:10:41.995023 4427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:41.996469 4496 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:10:41.997056 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap starting.
I20240501 02:10:42.005268 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4427
I20240501 02:10:42.005527 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:37069
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:45189
--webserver_port=45331
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:42.006764 4496 log.cc:824] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:42.008571 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:10:42.008836 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap complete.
I20240501 02:10:42.010934 4496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.011149 4496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fdf156c108453184292c3e12f9ca0c, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.011288 4496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.011358 4496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:42.011396 4496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:42.011446 4496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:10:42.012393 4496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.012492 4496 leader_election.cc:302] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79fdf156c108453184292c3e12f9ca0c; no voters:
I20240501 02:10:42.012660 4496 leader_election.cc:288] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 02:10:42.012737 4500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 02:10:42.012912 4496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:42.012928 4500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 LEADER]: Becoming Leader. State: Replica: 79fdf156c108453184292c3e12f9ca0c, State: Running, Role: LEADER
I20240501 02:10:42.013018 4500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.013324 4501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "79fdf156c108453184292c3e12f9ca0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } } }
I20240501 02:10:42.013358 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79fdf156c108453184292c3e12f9ca0c. Latest consensus state: current_term: 2 leader_uuid: "79fdf156c108453184292c3e12f9ca0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } } }
I20240501 02:10:42.013404 4501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:42.013418 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:42.013631 4507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:42.014386 4507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:42.014482 4507 catalog_manager.cc:1259] Loaded cluster ID: 398ca7e61f204fe6ac0c4459d95cd17e
I20240501 02:10:42.014519 4507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:10:42.014791 4507 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20240501 02:10:42.016407 4507 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20240501 02:10:42.020011 4507 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:42.020239 4507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Loaded TSK: 0
I20240501 02:10:42.020452 4507 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 02:10:42.118126 4498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:42.118394 4498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:42.118433 4498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:42.120601 4498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:42.120680 4498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:42.120721 4498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:10:42.123307 4498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:37069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=45331
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:42.123581 4498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:42.123801 4498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:42.128479 4523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.128937 4524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.129364 4526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:42.129832 4498 server_base.cc:1034] running on GCE node
I20240501 02:10:42.130041 4498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:42.130290 4498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:42.131453 4498 hybrid_clock.cc:648] HybridClock initialized: now 1714529442131441 us; error 28 us; skew 500 ppm
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:42.134410 4498 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:10:42.135016 4498 webserver.cc:460] Webserver started at http://127.22.12.65:45331/ using document root <none> and password file <none>
I20240501 02:10:42.135246 4498 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:42.135322 4498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:42.136976 4498 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:10:42.137758 4533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.137948 4498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.138021 4498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "7f3bb53c321540d8a2310ec9a5d05f17"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "87d5fc7a5bfb4d60581e94f7dbe1ddcb"
server_key_iv: "f94d3916ba3a42ad5a0bc488696de1f6"
server_key_version: "encryptionkey@0"
I20240501 02:10:42.138332 4498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:42.152941 4498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:42.155297 4498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:42.155406 4498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:42.155655 4498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:42.156106 4540 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 02:10:42.157212 4498 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 02:10:42.157269 4498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240501 02:10:42.157305 4498 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 02:10:42.157873 4498 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 02:10:42.157907 4498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 02:10:42.158001 4540 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap starting.
I20240501 02:10:42.165231 4498 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:37069
I20240501 02:10:42.165443 4647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:37069 every 8 connection(s)
I20240501 02:10:42.165671 4498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:10:42.169713 4540 log.cc:824] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:42.170809 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4498
I20240501 02:10:42.171087 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:43529
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:45189
--webserver_port=33049
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:42.172775 4540 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:10:42.173074 4540 tablet_bootstrap.cc:492] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap complete.
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:42.174153 4540 ts_tablet_manager.cc:1367] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.009s
I20240501 02:10:42.175395 4540 raft_consensus.cc:348] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.175702 4540 raft_consensus.cc:729] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3bb53c321540d8a2310ec9a5d05f17, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.175863 4540 consensus_queue.cc:260] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.176131 4540 ts_tablet_manager.cc:1398] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:10:42.177336 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.166556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:39523)
0501 02:10:42.167951 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.167981 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.168437 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.168961 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.168966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.169306 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:42.169592 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.169596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.170894 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.170896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.171405 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.171409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.171512 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.172124 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:42.172148 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:42.173644 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:42.175778 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:42.175783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:42.175786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:42.176037 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:42.176304 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:42.176305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:42.176305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:42.176385 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:42.176752 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:42.176758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:42.176911 (+ 153us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.177007 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.177162 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":85,"threads_started":1}
I20240501 02:10:42.177744 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.166650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:39523 (local address 127.22.12.126:45189)
0501 02:10:42.168601 (+ 1951us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.168610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.168628 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:10:42.168724 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.168729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.168767 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.168898 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:42.169716 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.170769 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.171800 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.172052 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.173841 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:42.173864 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:42.173865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:42.173878 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:42.175677 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.176136 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.176139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.176139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.176205 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.176468 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.176469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.176470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.176672 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:42.176773 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.177087 (+ 314us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.177209 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1914,"thread_start_us":69,"threads_started":1}
I20240501 02:10:42.178400 4648 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45189
I20240501 02:10:42.178721 4648 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:42.178948 4648 heartbeater.cc:511] Master 127.22.12.126:45189 requested a full tablet report, sending...
I20240501 02:10:42.179638 4460 ts_manager.cc:194] Registered new tserver with Master: 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069)
I20240501 02:10:42.180701 4460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:39523
I20240501 02:10:42.181711 4648 heartbeater.cc:503] Master 127.22.12.126:45189 was elected leader, sending a full tablet report...
I20240501 02:10:42.189246 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.180200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37069 (local address 127.0.0.1:51784)
0501 02:10:42.186810 (+ 6610us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.186834 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.186938 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.187140 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.187143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.187145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:42.187383 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.187385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.188285 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.188287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.188940 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.188943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.189013 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.189051 (+ 38us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.189101 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.189145 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6583,"thread_start_us":105,"threads_started":1}
I20240501 02:10:42.189738 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.180463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.22.12.65:37069)
0501 02:10:42.186737 (+ 6274us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.186745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.186934 (+ 189us) server_negotiation.cc:407] Connection header received
0501 02:10:42.187039 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.187041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.187055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.187127 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:42.187461 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.188196 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.189033 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.189535 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.189588 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.189625 (+ 37us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.189672 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6228,"thread_start_us":87,"threads_started":1}
I20240501 02:10:42.190188 4582 tablet_service.cc:1514] Processing DeleteTablet for tablet 443dc1afa3de4122a758f66fdd84d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:41 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:51784
I20240501 02:10:42.190732 4657 tablet_replica.cc:330] stopping tablet replica
I20240501 02:10:42.190826 4657 raft_consensus.cc:2229] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:42.190897 4657 raft_consensus.cc:2258] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:42.191195 4657 ts_tablet_manager.cc:1875] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:42.193094 4657 ts_tablet_manager.cc:1888] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:42.193161 4657 log.cc:1196] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/443dc1afa3de4122a758f66fdd84d829
I20240501 02:10:42.193384 4657 ts_tablet_manager.cc:1909] T 443dc1afa3de4122a758f66fdd84d829 P 7f3bb53c321540d8a2310ec9a5d05f17: Deleting consensus metadata
I20240501 02:10:42.193883 4447 catalog_manager.cc:4830] TS 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069): tablet 443dc1afa3de4122a758f66fdd84d829 (table test-table [id=116b202c7a304a1cbb51ff160dd1702c]) successfully deleted
W20240501 02:10:42.270299 4653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:42.270488 4653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:42.270509 4653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:42.271853 4653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:42.271899 4653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:42.271924 4653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:10:42.273442 4653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:43529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=33049
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:42.273629 4653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:42.273794 4653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:42.278214 4662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.278628 4663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.279037 4665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:42.279498 4653 server_base.cc:1034] running on GCE node
I20240501 02:10:42.279654 4653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:42.279861 4653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:42.281014 4653 hybrid_clock.cc:648] HybridClock initialized: now 1714529442281004 us; error 27 us; skew 500 ppm
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:42.283385 4653 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:10:42.283808 4653 webserver.cc:460] Webserver started at http://127.22.12.66:33049/ using document root <none> and password file <none>
I20240501 02:10:42.283991 4653 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:42.284049 4653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:42.285300 4653 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.285811 4672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.285972 4653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:42.286047 4653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "88a97f00646947d7ab43045491d3c2d0"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "bc9f999eeb28d8ce9a7794f0386943a4"
server_key_iv: "964324158960a942f589b3569a308964"
server_key_version: "encryptionkey@0"
I20240501 02:10:42.286325 4653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:42.313606 4653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:42.315492 4653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:42.315608 4653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:42.315826 4653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:42.316119 4653 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:42.316151 4653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.316181 4653 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:42.316196 4653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.321516 4653 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:43529
I20240501 02:10:42.321578 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:43529 every 8 connection(s)
I20240501 02:10:42.321871 4653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:42.329782 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.322565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:33515 (local address 127.22.12.126:45189)
0501 02:10:42.322647 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.322649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.322810 (+ 161us) server_negotiation.cc:407] Connection header received
0501 02:10:42.323275 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.323276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.323290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.323338 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:42.324067 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.324561 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.325094 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.325222 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.327391 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:42.327396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:42.327396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:42.327402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:42.328685 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.329037 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.329038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.329038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.329068 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.329245 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.329246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.329246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.329348 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:42.329412 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.329620 (+ 208us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.329699 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 02:10:42.329862 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.322556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:33515)
0501 02:10:42.322698 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.322725 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.323188 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.323392 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.323396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.323694 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:42.323994 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.323998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.324631 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.324631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.325014 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.325016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.325393 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.325923 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:42.325939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:42.327304 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:42.328765 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:42.328770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:42.328773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:42.328967 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:42.329119 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:42.329120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:42.329120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:42.329167 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:42.329405 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:42.329407 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:42.329513 (+ 106us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.329597 (+ 84us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.329702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20240501 02:10:42.330405 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4653
I20240501 02:10:42.330397 4786 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45189
I20240501 02:10:42.330492 4786 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:42.330664 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:41307
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:45189
--webserver_port=37957
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:42.330720 4786 heartbeater.cc:511] Master 127.22.12.126:45189 requested a full tablet report, sending...
I20240501 02:10:42.331151 4460 ts_manager.cc:194] Registered new tserver with Master: 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529)
I20240501 02:10:42.331542 4460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:33515
W20240501 02:10:42.409269 4789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:42.409463 4789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:42.409485 4789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:42.410825 4789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:42.410869 4789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:42.410895 4789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:10:42.412500 4789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=37957
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:42.412684 4789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:42.412882 4789 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:42.417246 4797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.417817 4795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.418190 4794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:42.418694 4789 server_base.cc:1034] running on GCE node
I20240501 02:10:42.418879 4789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:42.419085 4789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:42.420214 4789 hybrid_clock.cc:648] HybridClock initialized: now 1714529442420195 us; error 36 us; skew 500 ppm
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:42.426849 4789 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:10:42.427268 4789 webserver.cc:460] Webserver started at http://127.22.12.67:37957/ using document root <none> and password file <none>
I20240501 02:10:42.427431 4789 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:42.427480 4789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:42.428731 4789 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.429302 4804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.429498 4789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.429579 4789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "2cb90544166949c99087cec1f4f17b0a"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "1fb03b1bf4bc98947bf06c9b8df60501"
server_key_iv: "ddafa4371135f497be630823337752ff"
server_key_version: "encryptionkey@0"
I20240501 02:10:42.429877 4789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:42.446094 4789 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:42.448235 4789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:42.448338 4789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:42.448515 4789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:42.448791 4789 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:42.448822 4789 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.448843 4789 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:42.448853 4789 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.453969 4789 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:41307
I20240501 02:10:42.454032 4917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:41307 every 8 connection(s)
I20240501 02:10:42.454317 4789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:10:42.455539 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4789
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:42.462605 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.455895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.22.12.126:45189)
0501 02:10:42.456019 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.456020 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.456093 (+ 73us) server_negotiation.cc:407] Connection header received
0501 02:10:42.456156 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.456156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.456163 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.456220 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:42.456977 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.457820 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.458577 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.458748 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.459968 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:42.459973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:42.459974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:42.459980 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:42.461582 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.461904 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.461905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.461905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.461928 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.462059 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.462060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.462060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.462172 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:42.462298 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.462390 (+ 92us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.462474 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20240501 02:10:42.463449 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.455038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:51159 (local address 127.22.12.126:45189)
0501 02:10:42.455094 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.455096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.455254 (+ 158us) server_negotiation.cc:407] Connection header received
0501 02:10:42.455890 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.455891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.455904 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.455973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:42.456980 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.457819 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.458577 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.458761 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.460909 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:42.460914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:42.460914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:42.460920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:42.462121 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.462451 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.462452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.462453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.462479 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:42.462772 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:42.462773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:42.462773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:42.462878 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:42.462942 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.463278 (+ 336us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.463364 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:10:42.463541 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.455013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:51159)
0501 02:10:42.455144 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.455169 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.455754 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.456056 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.456061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.456473 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:42.456900 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.456906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.457903 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.457904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.458487 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.458488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.458955 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.459328 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:42.459348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:42.460811 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:42.462188 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:42.462193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:42.462195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:42.462373 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:42.462554 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:42.462555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:42.462556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:42.462620 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:42.463006 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:42.463010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:42.463150 (+ 140us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.463241 (+ 91us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.463377 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20240501 02:10:42.464030 4918 heartbeater.cc:347] Connected to a master server at 127.22.12.126:45189
I20240501 02:10:42.464112 4918 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:42.464282 4918 heartbeater.cc:511] Master 127.22.12.126:45189 requested a full tablet report, sending...
I20240501 02:10:42.464625 4460 ts_manager.cc:194] Registered new tserver with Master: 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
I20240501 02:10:42.464957 4460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:51159
I20240501 02:10:42.474500 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:10:42.477048 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.474931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.22.12.126:45189)
0501 02:10:42.475002 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.475004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.475069 (+ 65us) server_negotiation.cc:407] Connection header received
0501 02:10:42.475150 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.475151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.475183 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.475249 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:10:42.475517 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.475987 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.476555 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.476684 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.476726 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:42.476810 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:42.476878 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.476945 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.476970 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 02:10:42.477324 4460 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:10:42.478785 4460 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:10:42.481288 4582 tablet_service.cc:1467] Processing CreateTablet for tablet fafdc0d9a30447ab808fca83ca1fe38b (DEFAULT_TABLE table=test-table [id=54fa107baf034656a2ecb489c53c971b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:42.481508 4582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafdc0d9a30447ab808fca83ca1fe38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:42.483125 4540 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap starting.
I20240501 02:10:42.483489 4540 tablet_bootstrap.cc:654] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:42.484093 4540 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: No bootstrap required, opened a new log
I20240501 02:10:42.484143 4540 ts_tablet_manager.cc:1367] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.484221 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.481040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43529 (local address 127.0.0.1:50836)
0501 02:10:42.481269 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.481280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.481321 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.481795 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.481796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.481801 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:42.481985 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.481987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.483141 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.483142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.483962 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.483966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.484045 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.484069 (+ 24us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.484101 (+ 32us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.484146 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20240501 02:10:42.484244 4540 raft_consensus.cc:348] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.484300 4540 raft_consensus.cc:374] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:42.484318 4540 raft_consensus.cc:729] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3bb53c321540d8a2310ec9a5d05f17, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.484375 4540 consensus_queue.cc:260] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.484474 4540 ts_tablet_manager.cc:1398] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.485256 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.481414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.22.12.66:43529)
0501 02:10:42.481616 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.481630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.481640 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:10:42.481677 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.481679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.481693 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.481764 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:42.482087 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.483039 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.484101 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.484651 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.485103 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.485140 (+ 37us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.485190 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20240501 02:10:42.485884 4720 tablet_service.cc:1467] Processing CreateTablet for tablet fafdc0d9a30447ab808fca83ca1fe38b (DEFAULT_TABLE table=test-table [id=54fa107baf034656a2ecb489c53c971b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:42.486173 4720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafdc0d9a30447ab808fca83ca1fe38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:42.486393 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.481129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41307 (local address 127.0.0.1:38546)
0501 02:10:42.481281 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.481292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.481351 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.483712 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.483713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.483716 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:42.483891 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.483893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.485593 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.485595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.486186 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.486189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.486253 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.486256 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.486283 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.486320 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20240501 02:10:42.486968 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.483174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.22.12.67:41307)
0501 02:10:42.483445 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.483456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.483468 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:10:42.483512 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.483514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.483533 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.483625 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:42.484579 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.485509 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.486293 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.486805 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.486853 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.486880 (+ 27us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.486911 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
I20240501 02:10:42.487500 4852 tablet_service.cc:1467] Processing CreateTablet for tablet fafdc0d9a30447ab808fca83ca1fe38b (DEFAULT_TABLE table=test-table [id=54fa107baf034656a2ecb489c53c971b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:42.487752 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafdc0d9a30447ab808fca83ca1fe38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:42.489001 4928 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Bootstrap starting.
I20240501 02:10:42.489579 4928 tablet_bootstrap.cc:654] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:42.489836 4928 log.cc:824] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:42.490213 4930 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Bootstrap starting.
I20240501 02:10:42.490765 4930 tablet_bootstrap.cc:654] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:42.490804 4928 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: No bootstrap required, opened a new log
I20240501 02:10:42.490865 4928 ts_tablet_manager.cc:1367] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:10:42.491003 4930 log.cc:824] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:42.491742 4930 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: No bootstrap required, opened a new log
I20240501 02:10:42.491796 4930 ts_tablet_manager.cc:1367] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:10:42.493058 4928 raft_consensus.cc:348] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.493191 4928 raft_consensus.cc:374] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:42.493207 4928 raft_consensus.cc:729] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a97f00646947d7ab43045491d3c2d0, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.493300 4928 consensus_queue.cc:260] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.493424 4930 raft_consensus.cc:348] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.493552 4930 raft_consensus.cc:374] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:42.493577 4930 raft_consensus.cc:729] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb90544166949c99087cec1f4f17b0a, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.493675 4786 heartbeater.cc:503] Master 127.22.12.126:45189 was elected leader, sending a full tablet report...
I20240501 02:10:42.493690 4930 consensus_queue.cc:260] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.493929 4930 ts_tablet_manager.cc:1398] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:10:42.493932 4928 ts_tablet_manager.cc:1398] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:42.494048 4918 heartbeater.cc:503] Master 127.22.12.126:45189 was elected leader, sending a full tablet report...
I20240501 02:10:42.500674 4654 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:42.500790 4654 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.501045 4654 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
I20240501 02:10:42.503986 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.501222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43529 (local address 127.22.12.65:51471)
0501 02:10:42.501279 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.501291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.501336 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.501598 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.501599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.501601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:42.501834 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.501836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.502795 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.502796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.503744 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.503746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.503818 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.503820 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.503852 (+ 32us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.503885 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240501 02:10:42.503986 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.501322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41307 (local address 127.22.12.65:53869)
0501 02:10:42.501508 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.501516 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.501564 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.501675 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:42.501676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:42.501677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:42.501857 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.501858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.502795 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.502796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:42.503739 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:42.503743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.503816 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.503820 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:10:42.503852 (+ 32us) client_negotiation.cc:240] Negotiation successful
0501 02:10:42.503885 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":119,"threads_started":1}
I20240501 02:10:42.504415 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.501388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:53869 (local address 127.22.12.67:41307)
0501 02:10:42.501430 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.501432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.501572 (+ 140us) server_negotiation.cc:407] Connection header received
0501 02:10:42.501605 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.501606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.501614 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.501659 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:42.501912 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.502710 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.503828 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.504300 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.504307 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.504326 (+ 19us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.504357 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:10:42.504415 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.501416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:51471 (local address 127.22.12.66:43529)
0501 02:10:42.501477 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.501479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.501486 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:10:42.501510 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.501511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.501518 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.501574 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:42.501906 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.502710 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.503834 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.504287 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.504296 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.504319 (+ 23us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.504348 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 02:10:42.504734 4740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdc0d9a30447ab808fca83ca1fe38b" candidate_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a97f00646947d7ab43045491d3c2d0" is_pre_election: true
I20240501 02:10:42.504751 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdc0d9a30447ab808fca83ca1fe38b" candidate_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb90544166949c99087cec1f4f17b0a" is_pre_election: true
I20240501 02:10:42.504863 4872 raft_consensus.cc:2454] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f3bb53c321540d8a2310ec9a5d05f17 in term 0.
I20240501 02:10:42.504871 4740 raft_consensus.cc:2454] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f3bb53c321540d8a2310ec9a5d05f17 in term 0.
I20240501 02:10:42.505074 4535 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f3bb53c321540d8a2310ec9a5d05f17, 88a97f00646947d7ab43045491d3c2d0; no voters:
I20240501 02:10:42.505203 4654 raft_consensus.cc:2781] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:10:42.505271 4654 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:10:42.505297 4654 raft_consensus.cc:3037] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:42.506193 4654 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.506315 4654 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 1 election: Requested vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
I20240501 02:10:42.506603 4740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdc0d9a30447ab808fca83ca1fe38b" candidate_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a97f00646947d7ab43045491d3c2d0"
I20240501 02:10:42.506603 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdc0d9a30447ab808fca83ca1fe38b" candidate_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb90544166949c99087cec1f4f17b0a"
I20240501 02:10:42.506692 4740 raft_consensus.cc:3037] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:42.506702 4872 raft_consensus.cc:3037] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:42.507555 4872 raft_consensus.cc:2454] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f3bb53c321540d8a2310ec9a5d05f17 in term 1.
I20240501 02:10:42.507553 4740 raft_consensus.cc:2454] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f3bb53c321540d8a2310ec9a5d05f17 in term 1.
I20240501 02:10:42.507783 4535 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb90544166949c99087cec1f4f17b0a, 7f3bb53c321540d8a2310ec9a5d05f17; no voters:
I20240501 02:10:42.507908 4654 raft_consensus.cc:2781] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:42.508054 4654 raft_consensus.cc:686] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3bb53c321540d8a2310ec9a5d05f17, State: Running, Role: LEADER
I20240501 02:10:42.508129 4654 consensus_queue.cc:237] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.508921 4460 catalog_manager.cc:5484] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } health_report { overall_health: HEALTHY } } }
I20240501 02:10:42.513626 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.511683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.22.12.65:37069)
0501 02:10:42.511733 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:42.511735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:42.511743 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:10:42.511799 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:42.511800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:42.511816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:42.511876 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:10:42.512088 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.512576 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:42.513158 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:42.513308 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:42.513361 (+ 53us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:42.513447 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:42.513504 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:42.513557 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 02:10:42.513576 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:10:42.515030 4872 raft_consensus.cc:1261] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Refusing update from remote peer 7f3bb53c321540d8a2310ec9a5d05f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:42.515041 4740 raft_consensus.cc:1261] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Refusing update from remote peer 7f3bb53c321540d8a2310ec9a5d05f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:42.515304 4654 consensus_queue.cc:1035] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:42.515441 4935 consensus_queue.cc:1035] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:42.517510 4938 mvcc.cc:205] Tried to move back new op lower bound from 7022712596539441152 to 7022712596513857536. Current Snapshot: MvccSnapshot[applied={T|T < 7022712596539441152}]
I20240501 02:10:42.518031 4939 mvcc.cc:205] Tried to move back new op lower bound from 7022712596539441152 to 7022712596513857536. Current Snapshot: MvccSnapshot[applied={T|T < 7022712596539441152 or (T in {7022712596539441152})}]
I20240501 02:10:42.518489 4940 mvcc.cc:205] Tried to move back new op lower bound from 7022712596539441152 to 7022712596513857536. Current Snapshot: MvccSnapshot[applied={T|T < 7022712596539441152 or (T in {7022712596539441152})}]
I20240501 02:10:42.519784 4460 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60178:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:42.519881 4460 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60178:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:42.520527 4460 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Sending DeleteTablet for 3 replicas of tablet fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:42.520845 4720 tablet_service.cc:1514] Processing DeleteTablet for tablet fafdc0d9a30447ab808fca83ca1fe38b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:42 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:50836
I20240501 02:10:42.520884 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet fafdc0d9a30447ab808fca83ca1fe38b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:42 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:38546
I20240501 02:10:42.521044 4582 tablet_service.cc:1514] Processing DeleteTablet for tablet fafdc0d9a30447ab808fca83ca1fe38b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:42 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:51784
I20240501 02:10:42.521055 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 4498
I20240501 02:10:42.521159 4948 tablet_replica.cc:330] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: stopping tablet replica
I20240501 02:10:42.521227 4948 raft_consensus.cc:2229] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:42.521302 4948 raft_consensus.cc:2258] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:42.521641 4947 tablet_replica.cc:330] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: stopping tablet replica
I20240501 02:10:42.521728 4947 raft_consensus.cc:2229] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:42.521817 4947 raft_consensus.cc:2258] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:42.522208 4947 ts_tablet_manager.cc:1875] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:42.522680 4948 ts_tablet_manager.cc:1875] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:42.523919 4947 ts_tablet_manager.cc:1888] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:42.524008 4947 log.cc:1196] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:42.524076 4948 ts_tablet_manager.cc:1888] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:42.524143 4948 log.cc:1196] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:42.524276 4947 ts_tablet_manager.cc:1909] T fafdc0d9a30447ab808fca83ca1fe38b P 88a97f00646947d7ab43045491d3c2d0: Deleting consensus metadata
I20240501 02:10:42.524395 4948 ts_tablet_manager.cc:1909] T fafdc0d9a30447ab808fca83ca1fe38b P 2cb90544166949c99087cec1f4f17b0a: Deleting consensus metadata
I20240501 02:10:42.524753 4445 catalog_manager.cc:4830] TS 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): tablet fafdc0d9a30447ab808fca83ca1fe38b (table test-table [id=54fa107baf034656a2ecb489c53c971b]) successfully deleted
I20240501 02:10:42.524842 4445 catalog_manager.cc:4830] TS 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): tablet fafdc0d9a30447ab808fca83ca1fe38b (table test-table [id=54fa107baf034656a2ecb489c53c971b]) successfully deleted
W20240501 02:10:42.527379 4447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:37069 (error 108)
I20240501 02:10:42.527882 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.527748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37069 (local address 127.0.0.1:51806)
0501 02:10:42.527808 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.527844 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20240501 02:10:42.527911 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 4653
W20240501 02:10:42.528012 4447 catalog_manager.cc:4573] TS 7f3bb53c321540d8a2310ec9a5d05f17 (127.22.12.65:37069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fafdc0d9a30447ab808fca83ca1fe38b: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37069: connect: Connection refused (error 111)
I20240501 02:10:42.532789 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 4789
I20240501 02:10:42.537252 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 4427
I20240501 02:10:42.542110 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45189
--webserver_interface=127.22.12.126
--webserver_port=33367
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:45189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:42.621187 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:42.621402 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:42.621425 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:42.622782 4950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:42.622836 4950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:42.622849 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:42.622860 4950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:42.622870 4950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:42.624435 4950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:45189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:42.624667 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:42.624836 4950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:42.629251 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.629590 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.630105 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:42.630578 4950 server_base.cc:1034] running on GCE node
I20240501 02:10:42.630868 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:42.631116 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:42.632256 4950 hybrid_clock.cc:648] HybridClock initialized: now 1714529442632237 us; error 33 us; skew 500 ppm
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:42.635179 4950 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:42.635584 4950 webserver.cc:460] Webserver started at http://127.22.12.126:33367/ using document root <none> and password file <none>
I20240501 02:10:42.635787 4950 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:42.635830 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:42.637151 4950 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.637805 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.638058 4950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.638144 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "79fdf156c108453184292c3e12f9ca0c"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "d53464c5ba584eaaeac7b3bde102c0b1"
server_key_iv: "eb82ec461ebe1b6c3e6ad1fd35c75db1"
server_key_version: "encryptionkey@0"
I20240501 02:10:42.638453 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:42.649348 4950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:42.651088 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:42.651202 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:42.654704 4950 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:45189
I20240501 02:10:42.654757 5017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:45189 every 8 connection(s)
I20240501 02:10:42.655077 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:42.656455 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 4950
I20240501 02:10:42.656462 5018 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:10:42.656698 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:37069
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:45189
--webserver_port=45331
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:10:42.657081 5018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap starting.
I20240501 02:10:42.666270 5018 log.cc:824] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:42.668586 5018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:10:42.668862 5018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c: Bootstrap complete.
I20240501 02:10:42.670960 5018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.671162 5018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fdf156c108453184292c3e12f9ca0c, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.671290 5018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.671365 5018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:42.671391 5018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:42.671435 5018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 2 FOLLOWER]: Advancing to term 3
I20240501 02:10:42.672366 5018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.672461 5018 leader_election.cc:302] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79fdf156c108453184292c3e12f9ca0c; no voters:
I20240501 02:10:42.672631 5018 leader_election.cc:288] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [CANDIDATE]: Term 3 election: Requested vote from peers
I20240501 02:10:42.672689 5022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 02:10:42.672816 5022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [term 3 LEADER]: Becoming Leader. State: Replica: 79fdf156c108453184292c3e12f9ca0c, State: Running, Role: LEADER
I20240501 02:10:42.672878 5018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:42.672894 5022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } }
I20240501 02:10:42.673161 5024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "79fdf156c108453184292c3e12f9ca0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } } }
I20240501 02:10:42.673247 5024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:42.673156 5022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79fdf156c108453184292c3e12f9ca0c. Latest consensus state: current_term: 3 leader_uuid: "79fdf156c108453184292c3e12f9ca0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79fdf156c108453184292c3e12f9ca0c" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 45189 } } }
I20240501 02:10:42.673588 5022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79fdf156c108453184292c3e12f9ca0c [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:42.673789 5027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:42.674479 5027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:42.674623 5027 catalog_manager.cc:1259] Loaded cluster ID: 398ca7e61f204fe6ac0c4459d95cd17e
I20240501 02:10:42.674785 5027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20240501 02:10:42.674996 5027 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6c8663adcd google::LogMessage::Fail() at ??:0
@ 0x7f6c8663eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6c8663a7cc google::LogMessage::Flush() at ??:0
@ 0x7f6c8663bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6c88e12af6 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6c88e1d294 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6c87734a08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6c877293a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6c87a4c6db start_thread at ??:0
@ 0x7f6c85b8f71f clone at ??:0
W20240501 02:10:42.749109 5020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:42.749315 5020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:42.749339 5020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:42.750816 5020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:42.750862 5020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:42.750890 5020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:10:42.752534 5020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:37069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=45331
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:42.752738 5020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:42.752920 5020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:42.757920 5054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.758327 5057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.758752 5055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:42.759256 5020 server_base.cc:1034] running on GCE node
I20240501 02:10:42.759435 5020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:42.759651 5020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:42.760780 5020 hybrid_clock.cc:648] HybridClock initialized: now 1714529442760752 us; error 32 us; skew 500 ppm
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:42.763168 5020 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:10:42.763607 5020 webserver.cc:460] Webserver started at http://127.22.12.65:45331/ using document root <none> and password file <none>
I20240501 02:10:42.763792 5020 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:42.763832 5020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:42.765065 5020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.765673 5064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.765872 5020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.765969 5020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "7f3bb53c321540d8a2310ec9a5d05f17"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "87d5fc7a5bfb4d60581e94f7dbe1ddcb"
server_key_iv: "f94d3916ba3a42ad5a0bc488696de1f6"
server_key_version: "encryptionkey@0"
I20240501 02:10:42.766368 5020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:42.780689 5020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:42.783167 5020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:42.783282 5020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:42.783470 5020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:42.783927 5071 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 02:10:42.784783 5020 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 02:10:42.784847 5020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240501 02:10:42.784888 5020 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 02:10:42.785635 5020 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 02:10:42.785674 5020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 02:10:42.785732 5071 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap starting.
I20240501 02:10:42.793267 5020 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:37069
I20240501 02:10:42.793341 5178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:37069 every 8 connection(s)
I20240501 02:10:42.793701 5020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:10:42.796700 5071 log.cc:824] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:42.798702 5071 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:10:42.798946 5071 tablet_bootstrap.cc:492] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Bootstrap complete.
I20240501 02:10:42.799682 5071 ts_tablet_manager.cc:1367] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240501 02:10:42.800545 5071 raft_consensus.cc:348] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.800763 5071 raft_consensus.cc:729] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3bb53c321540d8a2310ec9a5d05f17, State: Initialized, Role: FOLLOWER
I20240501 02:10:42.800881 5071 consensus_queue.cc:260] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:42.801131 5071 ts_tablet_manager.cc:1398] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 02:10:42.801429 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5020
I20240501 02:10:42.801666 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:43529
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:45189
--webserver_port=33049
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:42.880908 5181 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:10:42.794363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:36427)
0501 02:10:42.794563 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.794595 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:42.795229 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:42.880736 (+ 85507us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
W20240501 02:10:42.881197 5179 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:45189 (0 consecutive failures): Network error: Failed to ping master at 127.22.12.126:45189: Client connection negotiation failed: client connection to 127.22.12.126:45189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 02:10:42.881561 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.881438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:34941)
0501 02:10:42.881491 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.881533 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 02:10:42.881700 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240501 02:10:42.881892 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.881797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:56819)
0501 02:10:42.881845 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.881871 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 02:10:42.881929 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:42.881968 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:42.881984 5179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 02:10:42.883445 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:42.883496 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:42.883522 5184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:10:42.885195 5184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:43529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=33049
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:42.885380 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:42.885541 5184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:42.890246 5189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.890411 5190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:42.890851 5192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:42.891286 5184 server_base.cc:1034] running on GCE node
I20240501 02:10:42.891453 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:42.891685 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:42.892818 5184 hybrid_clock.cc:648] HybridClock initialized: now 1714529442892800 us; error 35 us; skew 500 ppm
May 01 02:10:42 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:42.895193 5184 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:10:42.895609 5184 webserver.cc:460] Webserver started at http://127.22.12.66:33049/ using document root <none> and password file <none>
I20240501 02:10:42.895784 5184 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:42.895826 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:42.897022 5184 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:42.897526 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.897704 5184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.897778 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "88a97f00646947d7ab43045491d3c2d0"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "bc9f999eeb28d8ce9a7794f0386943a4"
server_key_iv: "964324158960a942f589b3569a308964"
server_key_version: "encryptionkey@0"
I20240501 02:10:42.898068 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:42.925076 5184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:42.926851 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:42.926965 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:42.927179 5184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:42.927466 5184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:42.927497 5184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.927526 5184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:42.927542 5184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:42.932633 5184 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:43529
I20240501 02:10:42.932696 5312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:43529 every 8 connection(s)
I20240501 02:10:42.932986 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:10:42.933930 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.933617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:34587)
0501 02:10:42.933756 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.933816 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20240501 02:10:42.934124 5313 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:45189 (0 consecutive failures): Network error: Failed to ping master at 127.22.12.126:45189: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
I20240501 02:10:42.934355 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.934276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:58337)
0501 02:10:42.934316 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.934338 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 02:10:42.934592 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:42.934501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:49035)
0501 02:10:42.934554 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:42.934574 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20240501 02:10:42.934725 5313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 02:10:42.936586 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5184
I20240501 02:10:42.936820 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:41307
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:45189
--webserver_port=37957
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:43.019351 5316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:43.019551 5316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:43.019572 5316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:43.020860 5316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:43.020898 5316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:43.020921 5316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:10:43.022470 5316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=37957
--tserver_master_addrs=127.22.12.126:45189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:43.022703 5316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:43.022877 5316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:43.027365 5321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:43.027585 5324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:43.028028 5322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:43.028477 5316 server_base.cc:1034] running on GCE node
I20240501 02:10:43.028692 5316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:43.028916 5316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:43.030045 5316 hybrid_clock.cc:648] HybridClock initialized: now 1714529443030027 us; error 38 us; skew 500 ppm
May 01 02:10:43 dist-test-slave-9tkh krb5kdc[3879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529443, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:43.032426 5316 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:10:43.032850 5316 webserver.cc:460] Webserver started at http://127.22.12.67:37957/ using document root <none> and password file <none>
I20240501 02:10:43.033022 5316 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:43.033066 5316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:43.034273 5316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:10:43.034857 5331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:43.035012 5316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:43.035092 5316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "2cb90544166949c99087cec1f4f17b0a"
format_stamp: "Formatted at 2024-05-01 02:10:41 on dist-test-slave-9tkh"
server_key: "1fb03b1bf4bc98947bf06c9b8df60501"
server_key_iv: "ddafa4371135f497be630823337752ff"
server_key_version: "encryptionkey@0"
I20240501 02:10:43.035396 5316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:43.058418 5316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:43.060359 5316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:43.060482 5316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:43.060695 5316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:43.061023 5316 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:43.061059 5316 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:43.061089 5316 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:43.061103 5316 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:43.066429 5316 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:41307
I20240501 02:10:43.066489 5444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:41307 every 8 connection(s)
I20240501 02:10:43.066824 5316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:10:43.067764 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:43.067449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:33969)
0501 02:10:43.067584 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:43.067646 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20240501 02:10:43.067957 5445 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:45189 (0 consecutive failures): Network error: Failed to ping master at 127.22.12.126:45189: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
I20240501 02:10:43.068207 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:43.068125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:43001)
0501 02:10:43.068164 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:43.068188 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20240501 02:10:43.068516 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:43.068431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:54725)
0501 02:10:43.068476 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:43.068496 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:10:43.068617 5445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 02:10:43.071476 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5316
I20240501 02:10:43.882582 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:43.882324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:48777)
0501 02:10:43.882442 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:43.882493 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20240501 02:10:43.935454 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:43.935060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:45319)
0501 02:10:43.935230 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:43.935264 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20240501 02:10:44.069321 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.068990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:58655)
0501 02:10:44.069127 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:44.069151 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20240501 02:10:44.460047 5452 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:44.460201 5452 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:44.460494 5452 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
May 01 02:10:44 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.66@KRBTEST.COM
May 01 02:10:44 dist-test-slave-9tkh krb5kdc[3879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529442, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.67@KRBTEST.COM
I20240501 02:10:44.469249 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.460779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:58029 (local address 127.22.12.66:43529)
0501 02:10:44.460931 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:44.460941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:44.460965 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:10:44.461402 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:44.461404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:44.461444 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:44.461569 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:44.462464 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463201 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:44.463753 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463977 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:44.466015 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:44.466024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:44.466024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:44.466041 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:44.467686 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:44.468161 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:44.468163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:44.468164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:44.468252 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:44.468477 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:44.468478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:44.468478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:44.468698 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:44.468778 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:44.469037 (+ 259us) server_negotiation.cc:299] Negotiation successful
0501 02:10:44.469148 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20240501 02:10:44.469242 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.460652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:43529 (local address 127.22.12.65:58029)
0501 02:10:44.460769 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:44.460779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:44.460844 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:44.461633 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:44.461636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:44.462077 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:44.462338 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:44.462343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463308 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:44.463309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:44.463664 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:44.463665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463769 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:44.464465 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:44.464468 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:44.465908 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:44.467775 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:44.467781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:44.467784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:44.468078 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:44.468331 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:44.468332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:44.468333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:44.468409 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:44.468765 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:44.468771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:44.468881 (+ 110us) client_negotiation.cc:769] Sending connection context
0501 02:10:44.468973 (+ 92us) client_negotiation.cc:240] Negotiation successful
0501 02:10:44.469130 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20240501 02:10:44.469274 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.460923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:38339 (local address 127.22.12.67:41307)
0501 02:10:44.461025 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:44.461040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:44.461053 (+ 13us) server_negotiation.cc:407] Connection header received
0501 02:10:44.461423 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:44.461424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:44.461455 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:44.461569 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:44.462462 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463233 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:44.463726 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463936 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:44.466235 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:44.466245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:44.466245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:44.466265 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:44.467892 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:44.468225 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:44.468228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:44.468228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:44.468285 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:44.468463 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:44.468463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:44.468464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:44.468704 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:44.468781 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:44.469034 (+ 253us) server_negotiation.cc:299] Negotiation successful
0501 02:10:44.469165 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
I20240501 02:10:44.469408 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.460814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:41307 (local address 127.22.12.65:38339)
0501 02:10:44.460912 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:44.460922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:44.460988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:44.461634 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:44.461636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:44.462077 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:44.462338 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:44.462343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463322 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:44.463323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:44.463653 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:44.463654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:44.463767 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:44.464390 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:44.464406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:44.466148 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:44.467973 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:44.467974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:44.467974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:44.468151 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:44.468356 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:44.468357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:44.468357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:44.468406 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:44.468772 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:44.468773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:44.468881 (+ 108us) client_negotiation.cc:769] Sending connection context
0501 02:10:44.468973 (+ 92us) client_negotiation.cc:240] Negotiation successful
0501 02:10:44.469217 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":110,"thread_start_us":43,"threads_started":1}
W20240501 02:10:44.470329 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
W20240501 02:10:44.470408 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:44.470432 5066 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f3bb53c321540d8a2310ec9a5d05f17; no voters: 2cb90544166949c99087cec1f4f17b0a, 88a97f00646947d7ab43045491d3c2d0
I20240501 02:10:44.470551 5452 raft_consensus.cc:2726] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:10:44.883141 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.883015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:54339)
0501 02:10:44.883062 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:44.883097 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20240501 02:10:44.936173 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:44.935953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:55497)
0501 02:10:44.936080 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:44.936128 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20240501 02:10:45.070053 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:45.069799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:41701)
0501 02:10:45.069946 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:45.069995 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20240501 02:10:45.883924 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:45.883691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:39083)
0501 02:10:45.883832 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:45.883874 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20240501 02:10:45.936939 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:45.936704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:59451)
0501 02:10:45.936846 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:45.936891 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20240501 02:10:46.070840 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:46.070589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:53477)
0501 02:10:46.070726 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:46.070764 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20240501 02:10:46.287660 5462 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:46.287768 5462 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:46.287941 5462 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
W20240501 02:10:46.288372 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
W20240501 02:10:46.288435 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:46.288450 5066 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f3bb53c321540d8a2310ec9a5d05f17; no voters: 2cb90544166949c99087cec1f4f17b0a, 88a97f00646947d7ab43045491d3c2d0
I20240501 02:10:46.288529 5462 raft_consensus.cc:2726] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:10:46.884635 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:46.884400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:32871)
0501 02:10:46.884528 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:46.884571 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20240501 02:10:46.937675 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:46.937447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:41441)
0501 02:10:46.937583 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:46.937627 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20240501 02:10:47.071496 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:47.071263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:58675)
0501 02:10:47.071382 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:47.071423 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20240501 02:10:47.885447 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:47.885185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:54995)
0501 02:10:47.885350 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:47.885395 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20240501 02:10:47.938350 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:47.938130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:50095)
0501 02:10:47.938271 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:47.938308 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20240501 02:10:48.025768 5468 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:48.025878 5468 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:48.026038 5468 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
W20240501 02:10:48.026414 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
W20240501 02:10:48.026504 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:48.026548 5066 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f3bb53c321540d8a2310ec9a5d05f17; no voters: 2cb90544166949c99087cec1f4f17b0a, 88a97f00646947d7ab43045491d3c2d0
I20240501 02:10:48.026628 5468 raft_consensus.cc:2726] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:10:48.072172 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:48.071940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:35323)
0501 02:10:48.072071 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:48.072108 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20240501 02:10:48.886257 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:48.886010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:34963)
0501 02:10:48.886160 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:48.886201 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20240501 02:10:48.939049 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:48.938810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:49477)
0501 02:10:48.938955 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:48.938998 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20240501 02:10:49.072822 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:49.072599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:45745)
0501 02:10:49.072730 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:49.072770 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20240501 02:10:49.887058 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:49.886816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:52137)
0501 02:10:49.886962 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:49.887005 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20240501 02:10:49.939733 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:49.939527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:41379)
0501 02:10:49.939646 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:49.939687 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20240501 02:10:50.073501 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:50.073272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:55057)
0501 02:10:50.073416 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:50.073453 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20240501 02:10:50.464304 5476 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:50.464399 5476 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:50.464529 5476 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
W20240501 02:10:50.464903 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
W20240501 02:10:50.464978 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:50.465003 5066 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f3bb53c321540d8a2310ec9a5d05f17; no voters: 2cb90544166949c99087cec1f4f17b0a, 88a97f00646947d7ab43045491d3c2d0
I20240501 02:10:50.465085 5476 raft_consensus.cc:2726] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:10:50.887813 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:50.887577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:55119)
0501 02:10:50.887723 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:50.887765 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20240501 02:10:50.940445 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:50.940197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:34165)
0501 02:10:50.940351 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:50.940396 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20240501 02:10:51.074204 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:51.073977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:52805)
0501 02:10:51.074110 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:51.074153 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20240501 02:10:51.888546 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:51.888322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:40807)
0501 02:10:51.888456 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:51.888498 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20240501 02:10:51.941107 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:51.940874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:33589)
0501 02:10:51.941011 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:51.941055 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20240501 02:10:52.074952 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:52.074706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:43683)
0501 02:10:52.074854 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:52.074900 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20240501 02:10:52.889400 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:52.889141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:56999)
0501 02:10:52.889306 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:52.889350 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20240501 02:10:52.941875 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:52.941581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:49979)
0501 02:10:52.941782 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:52.941825 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":65,"threads_started":1}
I20240501 02:10:53.075668 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:53.075456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:50933)
0501 02:10:53.075578 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:53.075618 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20240501 02:10:53.890136 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:53.889932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:56149)
0501 02:10:53.890057 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:53.890090 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20240501 02:10:53.942436 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:53.942248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:35811)
0501 02:10:53.942359 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:53.942390 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20240501 02:10:54.076375 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:54.076157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:39205)
0501 02:10:54.076277 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:54.076321 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20240501 02:10:54.890825 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:54.890606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:41035)
0501 02:10:54.890742 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:54.890774 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20240501 02:10:54.943189 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:54.942915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:34531)
0501 02:10:54.943075 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:54.943127 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20240501 02:10:55.077178 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:55.076929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:47215)
0501 02:10:55.077071 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:55.077127 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20240501 02:10:55.891530 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:55.891296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:41807)
0501 02:10:55.891447 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:55.891482 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20240501 02:10:55.943791 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:55.943607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:44045)
0501 02:10:55.943718 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:55.943746 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20240501 02:10:56.077942 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:56.077722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:42287)
0501 02:10:56.077850 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:56.077892 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20240501 02:10:56.892237 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:56.892019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:54749)
0501 02:10:56.892150 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:56.892188 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20240501 02:10:56.944458 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:56.944220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:43331)
0501 02:10:56.944369 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:56.944411 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20240501 02:10:57.078665 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:57.078399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:35813)
0501 02:10:57.078572 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:57.078611 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20240501 02:10:57.323606 5498 raft_consensus.cc:482] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:57.323709 5498 raft_consensus.cc:504] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a97f00646947d7ab43045491d3c2d0" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 43529 } } peers { permanent_uuid: "2cb90544166949c99087cec1f4f17b0a" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 41307 } } peers { permanent_uuid: "7f3bb53c321540d8a2310ec9a5d05f17" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37069 } }
I20240501 02:10:57.323859 5498 leader_election.cc:288] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529), 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307)
W20240501 02:10:57.324285 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88a97f00646947d7ab43045491d3c2d0 (127.22.12.66:43529): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
W20240501 02:10:57.324354 5066 leader_election.cc:341] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb90544166949c99087cec1f4f17b0a (127.22.12.67:41307): Not found: Tablet not found: fafdc0d9a30447ab808fca83ca1fe38b
I20240501 02:10:57.324374 5066 leader_election.cc:302] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f3bb53c321540d8a2310ec9a5d05f17; no voters: 2cb90544166949c99087cec1f4f17b0a, 88a97f00646947d7ab43045491d3c2d0
I20240501 02:10:57.324457 5498 raft_consensus.cc:2726] T fafdc0d9a30447ab808fca83ca1fe38b P 7f3bb53c321540d8a2310ec9a5d05f17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:10:57.892976 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:57.892751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.65:35217)
0501 02:10:57.892885 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:57.892923 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20240501 02:10:57.945240 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:57.944990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.66:52643)
0501 02:10:57.945135 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:57.945193 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20240501 02:10:58.078254 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5020
I20240501 02:10:58.080189 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:58.079985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:45189 (local address 127.22.12.67:48117)
0501 02:10:58.080104 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:58.080135 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20240501 02:10:58.083061 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5184
I20240501 02:10:58.087204 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5316
2024-05-01T02:10:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18014 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:10:58 dist-test-slave-9tkh krb5kdc[5504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:10:58 dist-test-slave-9tkh krb5kdc[5504](info): set up 2 sockets
May 01 02:10:58 dist-test-slave-9tkh krb5kdc[5504](info): commencing operation
krb5kdc: starting...
W20240501 02:10:58.974427 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:10:59.085333 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41739
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:59.163805 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:59.164005 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:59.164034 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:59.165364 5517 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240501 02:10:59.165411 5517 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240501 02:10:59.165427 5517 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240501 02:10:59.165441 5517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:10:59.165455 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:59.165472 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:59.165488 5517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:10:59.165505 5517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:10:59.167218 5517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:59.167436 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:59.167609 5517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:59.171486 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.171761 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.172120 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:59.172609 5517 server_base.cc:1034] running on GCE node
I20240501 02:10:59.172876 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:59.173120 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:59.174261 5517 hybrid_clock.cc:648] HybridClock initialized: now 1714529459174243 us; error 32 us; skew 500 ppm
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:59.176705 5517 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:10:59.177148 5517 webserver.cc:460] Webserver started at http://127.22.12.126:40103/ using document root <none> and password file <none>
I20240501 02:10:59.177317 5517 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:59.177357 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:59.177459 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:59.178462 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "bd10818eed9f458f81b6ee4230ca066a"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "2bf543d15b56c55f3b010f33daf02ea4"
server_key_iv: "dc06475182bbf62f0bb8bcd6a44c201e"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.178809 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "bd10818eed9f458f81b6ee4230ca066a"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "2bf543d15b56c55f3b010f33daf02ea4"
server_key_iv: "dc06475182bbf62f0bb8bcd6a44c201e"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.180027 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:59.180788 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.180970 5517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:10:59.181054 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "bd10818eed9f458f81b6ee4230ca066a"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "2bf543d15b56c55f3b010f33daf02ea4"
server_key_iv: "dc06475182bbf62f0bb8bcd6a44c201e"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.181114 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:59.191558 5517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:59.194260 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:59.194447 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:59.198729 5517 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41739
I20240501 02:10:59.198776 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41739 every 8 connection(s)
I20240501 02:10:59.199059 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:10:59.199419 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5517
I20240501 02:10:59.199524 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:10:59.199682 22577 external_mini_cluster.cc:1428] Setting key 01df69fb717cef75112b2519f0da048e
I20240501 02:10:59.199959 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:59.203047 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap starting.
I20240501 02:10:59.203677 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:59.203920 5585 log.cc:824] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:59.204572 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: No bootstrap required, opened a new log
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:59.206363 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:10:59.206483 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:59.206498 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd10818eed9f458f81b6ee4230ca066a, State: Initialized, Role: FOLLOWER
I20240501 02:10:59.206616 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:10:59.206712 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:10:59.206748 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:10:59.206794 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:59.207311 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:10:59.207409 5585 leader_election.cc:302] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd10818eed9f458f81b6ee4230ca066a; no voters:
I20240501 02:10:59.207548 5585 leader_election.cc:288] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:10:59.207630 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:59.207764 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 LEADER]: Becoming Leader. State: Replica: bd10818eed9f458f81b6ee4230ca066a, State: Running, Role: LEADER
I20240501 02:10:59.207796 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:10:59.207887 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:10:59.208199 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd10818eed9f458f81b6ee4230ca066a. Latest consensus state: current_term: 1 leader_uuid: "bd10818eed9f458f81b6ee4230ca066a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } } }
I20240501 02:10:59.208362 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:59.208586 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:10:59.209039 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:10:59.209538 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.200181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.22.12.126:41739)
0501 02:10:59.200329 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.200339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.200358 (+ 19us) server_negotiation.cc:407] Connection header received
0501 02:10:59.200775 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.200781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.200824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.200938 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:59.201884 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.202666 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.203413 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.203586 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.205806 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:59.205814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:59.205814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:59.205831 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:59.207528 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.208134 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.208138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.208138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.208209 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.208377 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.208378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.208378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.208630 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:59.208826 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.209287 (+ 461us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.209398 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20240501 02:10:59.209745 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd10818eed9f458f81b6ee4230ca066a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } } }
I20240501 02:10:59.209813 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: This master's current role is: LEADER
I20240501 02:10:59.214927 5599 catalog_manager.cc:1347] Generated new cluster ID: a8a4435457ab456180c374d8d26baa64
I20240501 02:10:59.214985 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:10:59.224062 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:10:59.224603 5599 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:10:59.228802 5599 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Generated new TSK 0
I20240501 02:10:59.229038 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:10:59.265666 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41739
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:59.351022 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:59.351219 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:59.351253 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:59.352586 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:59.352635 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:59.352669 5611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:10:59.354256 5611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:59.354476 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:59.354692 5611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:59.358605 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.359243 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.359612 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:59.359628 5611 server_base.cc:1034] running on GCE node
I20240501 02:10:59.359911 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:59.360173 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:59.361336 5611 hybrid_clock.cc:648] HybridClock initialized: now 1714529459361300 us; error 52 us; skew 500 ppm
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:59.364315 5611 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:10:59.364849 5611 webserver.cc:460] Webserver started at http://127.22.12.65:36673/ using document root <none> and password file <none>
I20240501 02:10:59.365060 5611 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:59.365114 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:59.365227 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:59.366451 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "e744b2014e89b444615fa04b316dda16"
server_key_iv: "3299022a611d7f45a2c3f816ce1b2df3"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.366811 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "e744b2014e89b444615fa04b316dda16"
server_key_iv: "3299022a611d7f45a2c3f816ce1b2df3"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.368160 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:59.368942 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.369102 5611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:10:59.369168 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "e744b2014e89b444615fa04b316dda16"
server_key_iv: "3299022a611d7f45a2c3f816ce1b2df3"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.369241 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:59.387315 5611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:59.389519 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:59.389621 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:59.389801 5611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:59.390064 5611 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:59.390092 5611 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.390113 5611 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:59.390129 5611 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.395283 5611 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38483
I20240501 02:10:59.395341 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38483 every 8 connection(s)
I20240501 02:10:59.395612 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:59.400800 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5611
I20240501 02:10:59.400913 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:10:59.401080 22577 external_mini_cluster.cc:1428] Setting key cd6e982b64a39e6e4b758a611b47f03c
I20240501 02:10:59.403837 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.396272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:45039 (local address 127.22.12.126:41739)
0501 02:10:59.396315 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.396317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.396488 (+ 171us) server_negotiation.cc:407] Connection header received
0501 02:10:59.396968 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.396970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.396979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.397041 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:59.398054 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.398596 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.399121 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.399268 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.401200 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:59.401207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:59.401207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:59.401215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:59.402598 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.403002 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.403002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.403002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.403019 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.403235 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.403237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.403238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.403358 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:59.403435 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.403661 (+ 226us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.403754 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:10:59.403864 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.396250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:45039)
0501 02:10:59.396398 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.396427 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.396866 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.397096 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.397099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.397383 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:59.397955 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.397960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.398661 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.398662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.399033 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.399034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.399134 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.399621 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:59.399637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:59.401074 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:59.402671 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:59.402677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:59.402681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:59.402935 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:59.403066 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:59.403067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:59.403067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:59.403143 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:59.403426 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:59.403429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:59.403535 (+ 106us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.403621 (+ 86us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.403723 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20240501 02:10:59.404353 5740 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41739
I20240501 02:10:59.404454 5740 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:59.404659 5740 heartbeater.cc:511] Master 127.22.12.126:41739 requested a full tablet report, sending...
I20240501 02:10:59.405182 5549 ts_manager.cc:194] Registered new tserver with Master: c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483)
I20240501 02:10:59.405826 5549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:45039
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:10:59.423007 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41739
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:59.499981 5745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:59.500171 5745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:59.500205 5745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:59.501508 5745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:59.501557 5745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:59.501590 5745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:10:59.503211 5745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:59.503434 5745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:59.503618 5745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:59.507618 5751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.507941 5753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.508329 5750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:59.508718 5745 server_base.cc:1034] running on GCE node
I20240501 02:10:59.508919 5745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:59.509132 5745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:59.510294 5745 hybrid_clock.cc:648] HybridClock initialized: now 1714529459510278 us; error 27 us; skew 500 ppm
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:59.512632 5745 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:10:59.513064 5745 webserver.cc:460] Webserver started at http://127.22.12.66:46667/ using document root <none> and password file <none>
I20240501 02:10:59.513244 5745 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:59.513284 5745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:59.513374 5745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:59.514371 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "3135be291ab34db5801da64c1ece1781"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "df58ebe72ed25a0508e1c8ece7fdfd3b"
server_key_iv: "655ed5312d990a33a645945cba206e60"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.514709 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "3135be291ab34db5801da64c1ece1781"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "df58ebe72ed25a0508e1c8ece7fdfd3b"
server_key_iv: "655ed5312d990a33a645945cba206e60"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.515817 5745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:10:59.516495 5760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.516690 5745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:10:59.516855 5745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "3135be291ab34db5801da64c1ece1781"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "df58ebe72ed25a0508e1c8ece7fdfd3b"
server_key_iv: "655ed5312d990a33a645945cba206e60"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.516934 5745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:59.526145 5745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:59.527907 5745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:59.528017 5745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:59.528215 5745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:59.528507 5745 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:59.528538 5745 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.528566 5745 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:59.528596 5745 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.533885 5745 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:44671
I20240501 02:10:59.533936 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:44671 every 8 connection(s)
I20240501 02:10:59.534210 5745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:10:59.537237 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5745
I20240501 02:10:59.537320 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:10:59.537467 22577 external_mini_cluster.cc:1428] Setting key f572c1cd04f8702f22cbe2c6cdd7d711
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:59.542665 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.534857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:55483 (local address 127.22.12.126:41739)
0501 02:10:59.534923 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.534926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.535076 (+ 150us) server_negotiation.cc:407] Connection header received
0501 02:10:59.535563 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.535564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.535575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.535658 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:59.536671 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.537164 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.537809 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.537983 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.540061 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:59.540065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:59.540066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:59.540073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:59.541514 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.541866 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.541868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.541868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.541895 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.542051 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.542052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.542052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.542178 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:59.542246 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.542460 (+ 214us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.542570 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:10:59.542693 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.534826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:55483)
0501 02:10:59.534965 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.534989 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.535444 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.535709 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.535713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.536009 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:59.536561 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.536566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.537270 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.537271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.537727 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.537730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.537863 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.538424 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:59.538440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:59.539973 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:59.541592 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:59.541597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:59.541599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:59.541798 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:59.541952 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:59.541953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:59.541953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:59.542001 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:59.542235 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:59.542237 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:59.542342 (+ 105us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.542423 (+ 81us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.542544 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20240501 02:10:59.543140 5874 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41739
I20240501 02:10:59.543246 5874 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:59.543478 5874 heartbeater.cc:511] Master 127.22.12.126:41739 requested a full tablet report, sending...
I20240501 02:10:59.543864 5549 ts_manager.cc:194] Registered new tserver with Master: 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671)
I20240501 02:10:59.544243 5549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:55483
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:10:59.560716 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41739
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:10:59.636945 5879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:10:59.637137 5879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:10:59.637158 5879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:10:59.638476 5879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:10:59.638521 5879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:10:59.638562 5879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:10:59.640146 5879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:10:59.640347 5879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:10:59.640528 5879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:10:59.644671 5885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.644932 5884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:10:59.645251 5887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:10:59.645742 5879 server_base.cc:1034] running on GCE node
I20240501 02:10:59.645932 5879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:10:59.646158 5879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:10:59.647364 5879 hybrid_clock.cc:648] HybridClock initialized: now 1714529459647306 us; error 36 us; skew 500 ppm
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:10:59.649820 5879 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:10:59.650243 5879 webserver.cc:460] Webserver started at http://127.22.12.67:42679/ using document root <none> and password file <none>
I20240501 02:10:59.650403 5879 fs_manager.cc:359] Metadata directory not provided
I20240501 02:10:59.650439 5879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:10:59.650560 5879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:10:59.651542 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "d9ed6edd16564293875fef463e3fd9d8"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "8f6a4ca473b8cd35636c1ff5b5136e2d"
server_key_iv: "20c94ee367e334e375557d828223a0d7"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.651825 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "d9ed6edd16564293875fef463e3fd9d8"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "8f6a4ca473b8cd35636c1ff5b5136e2d"
server_key_iv: "20c94ee367e334e375557d828223a0d7"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.652896 5879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:10:59.653477 5894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.653640 5879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:10:59.653730 5879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "d9ed6edd16564293875fef463e3fd9d8"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "8f6a4ca473b8cd35636c1ff5b5136e2d"
server_key_iv: "20c94ee367e334e375557d828223a0d7"
server_key_version: "encryptionkey@0"
I20240501 02:10:59.653805 5879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:10:59.671581 5879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:10:59.673588 5879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:10:59.673688 5879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:10:59.673863 5879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:10:59.674149 5879 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:10:59.674180 5879 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.674202 5879 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:10:59.674225 5879 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:10:59.679184 5879 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:40541
I20240501 02:10:59.679246 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:40541 every 8 connection(s)
I20240501 02:10:59.679502 5879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:10:59 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529459, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:10:59.684934 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 5879
I20240501 02:10:59.685034 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:10:59.685199 22577 external_mini_cluster.cc:1428] Setting key a540668e5992e71f494635df9f394407
I20240501 02:10:59.687597 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.680217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:42111 (local address 127.22.12.126:41739)
0501 02:10:59.680292 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.680295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.680381 (+ 86us) server_negotiation.cc:407] Connection header received
0501 02:10:59.680920 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.680921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.680931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.681004 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:59.681985 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.682475 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.682997 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.683103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.685189 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:59.685193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:59.685194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:59.685201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:59.686639 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.686985 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.686986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.686987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.687003 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.687130 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.687131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.687131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.687217 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:59.687266 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.687464 (+ 198us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.687529 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 02:10:59.687706 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.680137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:42111)
0501 02:10:59.680276 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.680302 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.680794 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.681056 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.681060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.681364 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:10:59.681877 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.681881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.682575 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.682576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.682942 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.682943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.683045 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.683560 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:10:59.683576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:10:59.685099 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:10:59.686719 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:59.686725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:59.686728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:59.686933 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:59.687045 (+ 112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:59.687045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:10:59.687045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:10:59.687093 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:59.687262 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:10:59.687264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:10:59.687366 (+ 102us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.687445 (+ 79us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.687548 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20240501 02:10:59.688143 6008 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41739
I20240501 02:10:59.688225 6008 heartbeater.cc:464] Registering TS with master...
I20240501 02:10:59.688400 6008 heartbeater.cc:511] Master 127.22.12.126:41739 requested a full tablet report, sending...
I20240501 02:10:59.688717 5549 ts_manager.cc:194] Registered new tserver with Master: d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
I20240501 02:10:59.689076 5549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:42111
I20240501 02:10:59.696991 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:10:59.703104 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.698274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.22.12.126:41739)
0501 02:10:59.698341 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.698343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.698418 (+ 75us) server_negotiation.cc:407] Connection header received
0501 02:10:59.698458 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.698459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.698467 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.698571 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:10:59.699212 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.699689 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.700198 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.700296 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.701211 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:10:59.701214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:10:59.701214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:10:59.701218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:10:59.702318 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.702605 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.702605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.702606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.702624 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:59.702720 (+ 96us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:10:59.702721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:10:59.702721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:10:59.702829 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:59.702896 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.702970 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.703034 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 02:10:59.704594 5549 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:10:59.706121 5549 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:10:59.711473 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.708114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:40541 (local address 127.0.0.1:42158)
0501 02:10:59.708418 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.708425 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.708469 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.708663 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.708666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.708668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:59.709106 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.709109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.710409 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.710410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.711168 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.711171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.711232 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.711278 (+ 46us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.711339 (+ 61us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.711389 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":60,"threads_started":1}
I20240501 02:10:59.711719 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.708092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38483 (local address 127.0.0.1:34192)
0501 02:10:59.708229 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.708254 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.708329 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.708861 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.708862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.708864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:59.709257 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.709260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.710472 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.710474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.711515 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.711517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.711586 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.711589 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.711616 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.711655 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20240501 02:10:59.711951 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.708216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.22.12.67:40541)
0501 02:10:59.708500 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.708509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.708519 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:10:59.708559 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.708561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.708576 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.708646 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:59.709197 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.710315 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.711257 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.711745 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.711795 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.711839 (+ 44us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.711879 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20240501 02:10:59.711987 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.708337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:44671 (local address 127.0.0.1:58554)
0501 02:10:59.708433 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.708443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.708493 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.709112 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.709113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.709115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:59.709326 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.709328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.710605 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.710606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.711517 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.711518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.711839 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.711842 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.711883 (+ 41us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.711922 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20240501 02:10:59.712617 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.708309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.22.12.65:38483)
0501 02:10:59.708706 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.708713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.708723 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:10:59.708765 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.708767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.708783 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.708861 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:59.709543 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.710372 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.711898 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.712423 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.712473 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.712504 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.712545 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":88,"threads_started":1}
I20240501 02:10:59.712548 5942 tablet_service.cc:1467] Processing CreateTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 (DEFAULT_TABLE table=test-table [id=fdac84cb409b485eb460fdecf9240702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:59.712782 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.708653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.22.12.66:44671)
0501 02:10:59.708890 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.708897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.708907 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:10:59.708944 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.708945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.708962 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.709037 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:59.709403 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.710504 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.712100 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.712600 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.712638 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.712668 (+ 30us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.712712 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20240501 02:10:59.712844 5942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a472d74f4fb4e4d8782d4b50eb32c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:59.713148 5674 tablet_service.cc:1467] Processing CreateTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 (DEFAULT_TABLE table=test-table [id=fdac84cb409b485eb460fdecf9240702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:59.713244 5808 tablet_service.cc:1467] Processing CreateTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 (DEFAULT_TABLE table=test-table [id=fdac84cb409b485eb460fdecf9240702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:10:59.713429 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a472d74f4fb4e4d8782d4b50eb32c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:59.713471 5808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a472d74f4fb4e4d8782d4b50eb32c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:10:59.715791 6022 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Bootstrap starting.
I20240501 02:10:59.716248 6024 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap starting.
I20240501 02:10:59.716361 6022 tablet_bootstrap.cc:654] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:59.716382 6023 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Bootstrap starting.
I20240501 02:10:59.716639 6022 log.cc:824] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:59.716790 6024 tablet_bootstrap.cc:654] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:59.716877 6023 tablet_bootstrap.cc:654] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Neither blocks nor log segments found. Creating new log.
I20240501 02:10:59.717079 6024 log.cc:824] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:59.717133 6023 log.cc:824] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Log is configured to *not* fsync() on all Append() calls
I20240501 02:10:59.717389 6022 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: No bootstrap required, opened a new log
I20240501 02:10:59.717432 6022 ts_tablet_manager.cc:1367] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:10:59.717739 6023 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: No bootstrap required, opened a new log
I20240501 02:10:59.717736 6024 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: No bootstrap required, opened a new log
I20240501 02:10:59.717782 6023 ts_tablet_manager.cc:1367] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:10:59.717789 6024 ts_tablet_manager.cc:1367] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:10:59.719552 6023 raft_consensus.cc:348] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.719676 6023 raft_consensus.cc:374] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:59.719586 6022 raft_consensus.cc:348] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.719699 6023 raft_consensus.cc:729] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed6edd16564293875fef463e3fd9d8, State: Initialized, Role: FOLLOWER
I20240501 02:10:59.719714 6022 raft_consensus.cc:374] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:59.719746 6022 raft_consensus.cc:729] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3135be291ab34db5801da64c1ece1781, State: Initialized, Role: FOLLOWER
I20240501 02:10:59.719844 6022 consensus_queue.cc:260] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.719885 6023 consensus_queue.cc:260] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.719851 6024 raft_consensus.cc:348] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.719976 6024 raft_consensus.cc:374] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:10:59.720000 6024 raft_consensus.cc:729] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d6f3e5f16c4b5d8c5b279ea33d3336, State: Initialized, Role: FOLLOWER
I20240501 02:10:59.720098 6022 ts_tablet_manager.cc:1398] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:10:59.720074 6024 consensus_queue.cc:260] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.720129 6023 ts_tablet_manager.cc:1398] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:10:59.720239 5874 heartbeater.cc:503] Master 127.22.12.126:41739 was elected leader, sending a full tablet report...
I20240501 02:10:59.720239 6008 heartbeater.cc:503] Master 127.22.12.126:41739 was elected leader, sending a full tablet report...
I20240501 02:10:59.720269 6024 ts_tablet_manager.cc:1398] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 02:10:59.720361 5740 heartbeater.cc:503] Master 127.22.12.126:41739 was elected leader, sending a full tablet report...
I20240501 02:10:59.720919 6029 raft_consensus.cc:482] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:10:59.721004 6029 raft_consensus.cc:504] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.721927 6029 leader_election.cc:288] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483), 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671)
I20240501 02:10:59.723369 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.721388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38483 (local address 127.22.12.67:55441)
0501 02:10:59.721479 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.721489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.721526 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.721645 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.721646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.721649 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:59.721858 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.721860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.722574 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.722576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.723192 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.723194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.723266 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.723269 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.723290 (+ 21us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.723315 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20240501 02:10:59.723824 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.721489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:55441 (local address 127.22.12.65:38483)
0501 02:10:59.721534 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.721536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.721546 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:10:59.721570 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.721571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.721579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.721627 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:59.721928 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.722473 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.723265 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.723726 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.723734 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.723754 (+ 20us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.723783 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:10:59.724052 5694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a472d74f4fb4e4d8782d4b50eb32c47" candidate_uuid: "d9ed6edd16564293875fef463e3fd9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" is_pre_election: true
I20240501 02:10:59.724151 5694 raft_consensus.cc:2454] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ed6edd16564293875fef463e3fd9d8 in term 0.
I20240501 02:10:59.724336 5898 leader_election.cc:302] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336, d9ed6edd16564293875fef463e3fd9d8; no voters:
I20240501 02:10:59.724478 6029 raft_consensus.cc:2781] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:10:59.724543 6029 raft_consensus.cc:482] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:10:59.724637 6029 raft_consensus.cc:3037] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:59.724725 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.721796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:44671 (local address 127.22.12.67:39693)
0501 02:10:59.722615 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.722625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:10:59.722679 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:10:59.722803 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:10:59.722803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:10:59.722805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:10:59.723011 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.723013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.723776 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.723777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:10:59.724427 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:10:59.724430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.724504 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.724508 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:10:59.724530 (+ 22us) client_negotiation.cc:240] Negotiation successful
0501 02:10:59.724562 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":107,"threads_started":1}
I20240501 02:10:59.725090 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.721834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:39693 (local address 127.22.12.66:44671)
0501 02:10:59.721872 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.721874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.722680 (+ 806us) server_negotiation.cc:407] Connection header received
0501 02:10:59.722740 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.722741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.722749 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.722798 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:10:59.723070 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.723678 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.724501 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.724976 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.724983 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.725005 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.725036 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:10:59.725344 5828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a472d74f4fb4e4d8782d4b50eb32c47" candidate_uuid: "d9ed6edd16564293875fef463e3fd9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3135be291ab34db5801da64c1ece1781" is_pre_election: true
I20240501 02:10:59.725464 5828 raft_consensus.cc:2454] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ed6edd16564293875fef463e3fd9d8 in term 0.
I20240501 02:10:59.725749 6029 raft_consensus.cc:504] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.725849 6029 leader_election.cc:288] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [CANDIDATE]: Term 1 election: Requested vote from peers c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483), 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671)
I20240501 02:10:59.726019 5694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a472d74f4fb4e4d8782d4b50eb32c47" candidate_uuid: "d9ed6edd16564293875fef463e3fd9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336"
I20240501 02:10:59.726080 5694 raft_consensus.cc:3037] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:59.726610 5828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a472d74f4fb4e4d8782d4b50eb32c47" candidate_uuid: "d9ed6edd16564293875fef463e3fd9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3135be291ab34db5801da64c1ece1781"
I20240501 02:10:59.726682 5828 raft_consensus.cc:3037] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:10:59.726773 5694 raft_consensus.cc:2454] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ed6edd16564293875fef463e3fd9d8 in term 1.
I20240501 02:10:59.726965 5898 leader_election.cc:302] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336, d9ed6edd16564293875fef463e3fd9d8; no voters:
I20240501 02:10:59.727077 6029 raft_consensus.cc:2781] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:10:59.727253 6029 raft_consensus.cc:686] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 1 LEADER]: Becoming Leader. State: Replica: d9ed6edd16564293875fef463e3fd9d8, State: Running, Role: LEADER
I20240501 02:10:59.727346 6029 consensus_queue.cc:237] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:10:59.727423 5828 raft_consensus.cc:2454] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ed6edd16564293875fef463e3fd9d8 in term 1.
I20240501 02:10:59.728112 5547 catalog_manager.cc:5484] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67). New cstate: current_term: 1 leader_uuid: "d9ed6edd16564293875fef463e3fd9d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } health_report { overall_health: HEALTHY } } }
I20240501 02:10:59.735731 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.733824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.22.12.67:40541)
0501 02:10:59.733881 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:10:59.733883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:10:59.733891 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:10:59.733926 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:10:59.733927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:10:59.733942 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:10:59.734006 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:10:59.734209 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.734721 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:59.735294 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:59.735420 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:59.735454 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:59.735536 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:59.735595 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 02:10:59.735642 (+ 47us) server_negotiation.cc:299] Negotiation successful
0501 02:10:59.735663 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:10:59.737416 5694 raft_consensus.cc:1261] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Refusing update from remote peer d9ed6edd16564293875fef463e3fd9d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:59.737416 5828 raft_consensus.cc:1261] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Refusing update from remote peer d9ed6edd16564293875fef463e3fd9d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:10:59.737776 6029 consensus_queue.cc:1035] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:59.737931 6032 consensus_queue.cc:1035] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:10:59.740089 6035 mvcc.cc:205] Tried to move back new op lower bound from 7022712667081924608 to 7022712667043966976. Current Snapshot: MvccSnapshot[applied={T|T < 7022712667081924608}]
I20240501 02:10:59.741155 6037 mvcc.cc:205] Tried to move back new op lower bound from 7022712667081924608 to 7022712667043966976. Current Snapshot: MvccSnapshot[applied={T|T < 7022712667081924608 or (T in {7022712667081924608})}]
I20240501 02:10:59.742017 5547 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51516:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:59.742116 5547 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51516:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:10:59.742885 5547 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Sending DeleteTablet for 3 replicas of tablet 2a472d74f4fb4e4d8782d4b50eb32c47
I20240501 02:10:59.743217 5674 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:59 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:34192
I20240501 02:10:59.743288 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5611
I20240501 02:10:59.743266 5808 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:59 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:58554
I20240501 02:10:59.743775 5942 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:59 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:42158
I20240501 02:10:59.743979 6046 tablet_replica.cc:330] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: stopping tablet replica
I20240501 02:10:59.744040 6045 tablet_replica.cc:330] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: stopping tablet replica
I20240501 02:10:59.744062 6046 raft_consensus.cc:2229] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:10:59.744145 6045 raft_consensus.cc:2229] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:10:59.744177 6046 raft_consensus.cc:2258] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:59.744264 6045 raft_consensus.cc:2258] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:10:59.744742 6045 ts_tablet_manager.cc:1875] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:59.744762 6046 ts_tablet_manager.cc:1875] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:10:59.746516 6045 ts_tablet_manager.cc:1888] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:59.746548 6046 ts_tablet_manager.cc:1888] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:10:59.746614 6046 log.cc:1196] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/2a472d74f4fb4e4d8782d4b50eb32c47
I20240501 02:10:59.746666 6045 log.cc:1196] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/2a472d74f4fb4e4d8782d4b50eb32c47
I20240501 02:10:59.746922 6046 ts_tablet_manager.cc:1909] T 2a472d74f4fb4e4d8782d4b50eb32c47 P d9ed6edd16564293875fef463e3fd9d8: Deleting consensus metadata
I20240501 02:10:59.747004 6045 ts_tablet_manager.cc:1909] T 2a472d74f4fb4e4d8782d4b50eb32c47 P 3135be291ab34db5801da64c1ece1781: Deleting consensus metadata
I20240501 02:10:59.747359 5534 catalog_manager.cc:4830] TS d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): tablet 2a472d74f4fb4e4d8782d4b50eb32c47 (table test-table [id=fdac84cb409b485eb460fdecf9240702]) successfully deleted
I20240501 02:10:59.747386 5536 catalog_manager.cc:4830] TS 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): tablet 2a472d74f4fb4e4d8782d4b50eb32c47 (table test-table [id=fdac84cb409b485eb460fdecf9240702]) successfully deleted
W20240501 02:10:59.749984 5536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:38483 (error 108)
I20240501 02:10:59.750485 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5745
I20240501 02:10:59.750505 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:10:59.750376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38483 (local address 127.0.0.1:34208)
0501 02:10:59.750431 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:59.750468 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240501 02:10:59.751804 5536 catalog_manager.cc:4573] TS c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a472d74f4fb4e4d8782d4b50eb32c47: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38483: connect: Connection refused (error 111)
I20240501 02:10:59.755385 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5879
I20240501 02:10:59.760219 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 5517
I20240501 02:11:19.765364 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41739
--webserver_interface=127.22.12.126
--webserver_port=40103
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:19.843389 6048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:19.843595 6048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:19.843627 6048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:19.845010 6048 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240501 02:11:19.845057 6048 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240501 02:11:19.845073 6048 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240501 02:11:19.845088 6048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:11:19.845103 6048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:19.845118 6048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:19.845135 6048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:11:19.845152 6048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:11:19.846880 6048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:19.847123 6048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:19.847304 6048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:19.854722 6055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:19.855031 6054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:19.855242 6057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:19.855628 6048 server_base.cc:1034] running on GCE node
I20240501 02:11:19.855823 6048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:19.856035 6048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:19.857153 6048 hybrid_clock.cc:648] HybridClock initialized: now 1714529479857145 us; error 26 us; skew 500 ppm
May 01 02:11:19 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529479, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:19.859596 6048 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:11:19.859992 6048 webserver.cc:460] Webserver started at http://127.22.12.126:40103/ using document root <none> and password file <none>
I20240501 02:11:19.860172 6048 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:19.860221 6048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:19.861449 6048 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:19.861987 6064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:19.862138 6048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:11:19.862221 6048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "bd10818eed9f458f81b6ee4230ca066a"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "2bf543d15b56c55f3b010f33daf02ea4"
server_key_iv: "dc06475182bbf62f0bb8bcd6a44c201e"
server_key_version: "encryptionkey@0"
I20240501 02:11:19.862584 6048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:19.887219 6048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:19.888928 6048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:19.889034 6048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:19.892773 6048 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41739
I20240501 02:11:19.892822 6116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41739 every 8 connection(s)
I20240501 02:11:19.893146 6048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:11:19.894837 6117 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:11:19.895476 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap starting.
I20240501 02:11:19.900235 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6048
I20240501 02:11:19.900503 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:38483
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:41739
--webserver_port=36673
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:11:19.904834 6117 log.cc:824] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:19.906730 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:11:19.906977 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap complete.
I20240501 02:11:19.908982 6117 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:19.909186 6117 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd10818eed9f458f81b6ee4230ca066a, State: Initialized, Role: FOLLOWER
I20240501 02:11:19.909322 6117 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:19.909390 6117 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:11:19.909435 6117 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:11:19.909482 6117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 1 FOLLOWER]: Advancing to term 2
I20240501 02:11:19.910399 6117 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:19.910496 6117 leader_election.cc:302] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd10818eed9f458f81b6ee4230ca066a; no voters:
I20240501 02:11:19.910676 6117 leader_election.cc:288] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 02:11:19.910727 6121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 02:11:19.910837 6121 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 LEADER]: Becoming Leader. State: Replica: bd10818eed9f458f81b6ee4230ca066a, State: Running, Role: LEADER
I20240501 02:11:19.910924 6117 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:11:19.910917 6121 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:19.911206 6121 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd10818eed9f458f81b6ee4230ca066a. Latest consensus state: current_term: 2 leader_uuid: "bd10818eed9f458f81b6ee4230ca066a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } } }
I20240501 02:11:19.911413 6121 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:19.911386 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bd10818eed9f458f81b6ee4230ca066a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } } }
I20240501 02:11:19.911623 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:19.911932 6125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:11:19.912680 6125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:11:19.912791 6125 catalog_manager.cc:1259] Loaded cluster ID: a8a4435457ab456180c374d8d26baa64
I20240501 02:11:19.913022 6125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:11:19.913923 6125 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:11:19.914091 6125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Loaded TSK: 0
I20240501 02:11:19.921280 6125 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Generated new TSK 1
I20240501 02:11:19.921404 6125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 02:11:19.993268 6119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:19.993484 6119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:19.993516 6119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:19.994890 6119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:19.994936 6119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:19.994971 6119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:11:19.996595 6119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:38483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=36673
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:19.996840 6119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:19.997026 6119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:20.001108 6145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:20.001539 6119 server_base.cc:1034] running on GCE node
W20240501 02:11:20.001910 6147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:20.002189 6144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:20.002446 6119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:20.002751 6119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:20.003932 6119 hybrid_clock.cc:648] HybridClock initialized: now 1714529480003918 us; error 34 us; skew 500 ppm
May 01 02:11:20 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:20.006880 6119 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:11:20.007316 6119 webserver.cc:460] Webserver started at http://127.22.12.65:36673/ using document root <none> and password file <none>
I20240501 02:11:20.007490 6119 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:20.007546 6119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:20.008817 6119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:20.009404 6154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.009570 6119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.009642 6119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "e744b2014e89b444615fa04b316dda16"
server_key_iv: "3299022a611d7f45a2c3f816ce1b2df3"
server_key_version: "encryptionkey@0"
I20240501 02:11:20.009918 6119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:20.021329 6119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:20.023630 6119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:20.023753 6119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:20.023962 6119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:20.024374 6161 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 02:11:20.025547 6119 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 02:11:20.025599 6119 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240501 02:11:20.025635 6119 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 02:11:20.026479 6119 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 02:11:20.026510 6119 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 02:11:20.026638 6161 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap starting.
I20240501 02:11:20.033006 6119 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38483
I20240501 02:11:20.033185 6268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38483 every 8 connection(s)
I20240501 02:11:20.033389 6119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:11:20.035650 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6119
I20240501 02:11:20.035946 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:44671
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:41739
--webserver_port=46667
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:11:20.038493 6161 log.cc:824] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:20.041683 6161 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:11:20.042013 6161 tablet_bootstrap.cc:492] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap complete.
May 01 02:11:20 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:20.043498 6161 ts_tablet_manager.cc:1367] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20240501 02:11:20.045106 6161 raft_consensus.cc:348] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:11:20.045403 6161 raft_consensus.cc:729] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d6f3e5f16c4b5d8c5b279ea33d3336, State: Initialized, Role: FOLLOWER
I20240501 02:11:20.045545 6161 consensus_queue.cc:260] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } } peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } }
I20240501 02:11:20.045805 6161 ts_tablet_manager.cc:1398] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:11:20.047061 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.034020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:35905)
0501 02:11:20.035907 (+ 1887us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.035947 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.036924 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.037605 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.037608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.037905 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:20.038192 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.038197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.039551 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.039552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.040062 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.040064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.040201 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.040843 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:20.040867 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:20.042449 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:20.045152 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:20.045159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:20.045162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:20.045448 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:20.045928 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:20.045928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:20.045929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:20.046019 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:20.046454 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:20.046460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:20.046633 (+ 173us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.046730 (+ 97us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.046882 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1833,"thread_start_us":66,"threads_started":1}
I20240501 02:11:20.047827 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.034123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:35905 (local address 127.22.12.126:41739)
0501 02:11:20.036321 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.036333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.036368 (+ 35us) server_negotiation.cc:407] Connection header received
0501 02:11:20.037217 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.037223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.037274 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.037415 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:20.038335 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.039445 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.040452 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.040728 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.042604 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:20.042614 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:20.042615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:20.042638 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:20.045020 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.045721 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.045724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.045724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.045799 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.046093 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.046094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.046094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.046366 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:20.047407 (+ 1041us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.047489 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.047635 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2146,"thread_start_us":69,"threads_started":1}
I20240501 02:11:20.048447 6269 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41739
I20240501 02:11:20.048537 6269 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:20.048758 6269 heartbeater.cc:511] Master 127.22.12.126:41739 requested a full tablet report, sending...
I20240501 02:11:20.049719 6081 ts_manager.cc:194] Registered new tserver with Master: c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483)
I20240501 02:11:20.051002 6081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:35905
I20240501 02:11:20.052124 6269 heartbeater.cc:503] Master 127.22.12.126:41739 was elected leader, sending a full tablet report...
I20240501 02:11:20.054986 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.050370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38483 (local address 127.0.0.1:34940)
0501 02:11:20.051477 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.051502 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.051572 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.051955 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.051958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.051960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:20.052271 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.052273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.053733 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.053734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.054695 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.054698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.054782 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.054812 (+ 30us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.054865 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.054904 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":80,"threads_started":1}
I20240501 02:11:20.055472 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.051289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.22.12.65:38483)
0501 02:11:20.051687 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.051695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.051703 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:11:20.051733 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.051734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.051845 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.052359 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:20.052417 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.053635 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.054781 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.055291 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.055336 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.055368 (+ 32us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.055408 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":91,"threads_started":1}
I20240501 02:11:20.055742 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a472d74f4fb4e4d8782d4b50eb32c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:10:59 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:34940
I20240501 02:11:20.056329 6278 tablet_replica.cc:330] stopping tablet replica
I20240501 02:11:20.056432 6278 raft_consensus.cc:2229] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:20.056504 6278 raft_consensus.cc:2258] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:20.056843 6278 ts_tablet_manager.cc:1875] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:20.058867 6278 ts_tablet_manager.cc:1888] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:20.058935 6278 log.cc:1196] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal/wals/2a472d74f4fb4e4d8782d4b50eb32c47
I20240501 02:11:20.059159 6278 ts_tablet_manager.cc:1909] T 2a472d74f4fb4e4d8782d4b50eb32c47 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Deleting consensus metadata
I20240501 02:11:20.059751 6068 catalog_manager.cc:4830] TS c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483): tablet 2a472d74f4fb4e4d8782d4b50eb32c47 (table test-table [id=fdac84cb409b485eb460fdecf9240702]) successfully deleted
W20240501 02:11:20.123764 6273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:20.124014 6273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:20.124039 6273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:20.126132 6273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:20.126204 6273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:20.126241 6273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:11:20.128247 6273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:44671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=46667
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:20.128469 6273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:20.128638 6273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:20.133056 6283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:20.133234 6286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:20.133405 6284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:20.133819 6273 server_base.cc:1034] running on GCE node
I20240501 02:11:20.133989 6273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:20.134202 6273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:20.135339 6273 hybrid_clock.cc:648] HybridClock initialized: now 1714529480135321 us; error 38 us; skew 500 ppm
May 01 02:11:20 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:20.137931 6273 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:11:20.138409 6273 webserver.cc:460] Webserver started at http://127.22.12.66:46667/ using document root <none> and password file <none>
I20240501 02:11:20.138633 6273 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:20.138693 6273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:20.140111 6273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:20.140746 6293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.140959 6273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:20.141048 6273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "3135be291ab34db5801da64c1ece1781"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "df58ebe72ed25a0508e1c8ece7fdfd3b"
server_key_iv: "655ed5312d990a33a645945cba206e60"
server_key_version: "encryptionkey@0"
I20240501 02:11:20.141345 6273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:20.159015 6273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:20.160881 6273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:20.161005 6273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:20.161216 6273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:20.161513 6273 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:20.161545 6273 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.161576 6273 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:20.161592 6273 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.166905 6273 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:44671
I20240501 02:11:20.166952 6406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:44671 every 8 connection(s)
I20240501 02:11:20.167268 6273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:11:20 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:20.172996 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6273
I20240501 02:11:20.173239 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:40541
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:41739
--webserver_port=42679
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:11:20.175571 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.167922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:50613 (local address 127.22.12.126:41739)
0501 02:11:20.168013 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.168015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.168175 (+ 160us) server_negotiation.cc:407] Connection header received
0501 02:11:20.168651 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.168652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.168668 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.168717 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:20.169706 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.170195 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.170769 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.170951 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.172919 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:20.172924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:20.172924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:20.172932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:20.174344 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.174762 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.174763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.174763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.174789 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.174964 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.174965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.174965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.175062 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:20.175117 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.175369 (+ 252us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.175475 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 02:11:20.175666 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.167895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:50613)
0501 02:11:20.168064 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.168094 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.168555 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.168804 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.168807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.169102 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:20.169604 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.169609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.170280 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.170281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.170695 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.170697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.170848 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.171401 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:20.171418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:20.172796 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:20.174428 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:20.174435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:20.174438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:20.174697 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:20.174840 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:20.174841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:20.174842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:20.174915 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:20.175113 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:20.175119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:20.175248 (+ 129us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.175344 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.175482 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20240501 02:11:20.176075 6407 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41739
I20240501 02:11:20.176169 6407 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:20.176375 6407 heartbeater.cc:511] Master 127.22.12.126:41739 requested a full tablet report, sending...
I20240501 02:11:20.176761 6081 ts_manager.cc:194] Registered new tserver with Master: 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671)
I20240501 02:11:20.177142 6081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:50613
W20240501 02:11:20.252218 6410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:20.252465 6410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:20.252501 6410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:20.253877 6410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:20.253926 6410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:20.253959 6410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:11:20.255631 6410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:40541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=42679
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:20.255875 6410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:20.256064 6410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:20.260874 6416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:20.261029 6415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:20.261469 6410 server_base.cc:1034] running on GCE node
W20240501 02:11:20.261709 6418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:20.261888 6410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:20.262082 6410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:20.263245 6410 hybrid_clock.cc:648] HybridClock initialized: now 1714529480263224 us; error 34 us; skew 500 ppm
May 01 02:11:20 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:20.265789 6410 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:11:20.266191 6410 webserver.cc:460] Webserver started at http://127.22.12.67:42679/ using document root <none> and password file <none>
I20240501 02:11:20.266345 6410 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:20.266388 6410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:20.267638 6410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:20.268154 6425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.268306 6410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:11:20.268374 6410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "d9ed6edd16564293875fef463e3fd9d8"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "8f6a4ca473b8cd35636c1ff5b5136e2d"
server_key_iv: "20c94ee367e334e375557d828223a0d7"
server_key_version: "encryptionkey@0"
I20240501 02:11:20.268659 6410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:20.283151 6410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:20.285351 6410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:20.285465 6410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:20.285676 6410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:20.285981 6410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:20.286013 6410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.286036 6410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:20.286057 6410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.291254 6410 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:40541
I20240501 02:11:20.291309 6538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:40541 every 8 connection(s)
I20240501 02:11:20.291600 6410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:11:20 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529480, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:20.297973 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6410
I20240501 02:11:20.300693 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.292382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:35477 (local address 127.22.12.126:41739)
0501 02:11:20.292451 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.292452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.292626 (+ 174us) server_negotiation.cc:407] Connection header received
0501 02:11:20.293096 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.293098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.293109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.293180 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:20.294181 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.294764 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.295418 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.295556 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.297552 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:20.297559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:20.297559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:20.297567 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:20.299349 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.299776 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.299778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.299778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.299802 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.300018 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.300019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.300020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.300114 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:20.300176 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.300499 (+ 323us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.300597 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 02:11:20.300734 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.292315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:35477)
0501 02:11:20.292530 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.292554 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.292986 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.293250 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.293254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.293557 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:20.294071 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.294076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.294864 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.294865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.295327 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.295330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.295455 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.296004 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:20.296023 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:20.297441 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:20.299448 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:20.299454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:20.299457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:20.299693 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:20.299858 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:20.299858 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:20.299859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:20.299940 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:20.300179 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:20.300195 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:20.300342 (+ 147us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.300460 (+ 118us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.300582 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20240501 02:11:20.301214 6539 heartbeater.cc:347] Connected to a master server at 127.22.12.126:41739
I20240501 02:11:20.301318 6539 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:20.301555 6539 heartbeater.cc:511] Master 127.22.12.126:41739 requested a full tablet report, sending...
I20240501 02:11:20.301971 6081 ts_manager.cc:194] Registered new tserver with Master: d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
I20240501 02:11:20.302347 6081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:35477
I20240501 02:11:20.304435 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.298570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.22.12.126:41739)
0501 02:11:20.298714 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.298716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.298725 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:11:20.298748 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.298749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.298759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.298815 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:20.299674 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.300375 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.301303 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.301457 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.302262 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:20.302266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:20.302267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:20.302273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:20.303658 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.303942 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.303943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.303943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.303960 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.304063 (+ 103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.304063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.304064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.304142 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:20.304252 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.304300 (+ 48us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.304366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20240501 02:11:20.305189 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:11:20.307518 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.305488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.22.12.126:41739)
0501 02:11:20.305536 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.305538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.305673 (+ 135us) server_negotiation.cc:407] Connection header received
0501 02:11:20.305726 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.305727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.305749 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.305800 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:11:20.306025 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.306502 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.307112 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.307237 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.307273 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:11:20.307313 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0501 02:11:20.307455 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49642: token expired
Metrics: {"server-negotiator.queue_time_us":29}
W20240501 02:11:20.307570 6543 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49642: token expired
W20240501 02:11:20.307672 6013 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.12.126:41739)
I20240501 02:11:20.312844 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.307872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.22.12.126:41739)
0501 02:11:20.307912 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.307914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.307937 (+ 23us) server_negotiation.cc:407] Connection header received
0501 02:11:20.307952 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.307953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.307961 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.307996 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:20.308481 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.308898 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.309381 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.309523 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.310486 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:20.310489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:20.310489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:20.310495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:20.311984 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.312322 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.312323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.312323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.312346 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:20.312460 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:20.312461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:20.312461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:20.312565 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:20.312631 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.312709 (+ 78us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.312778 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:11:20.314054 6081 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:11:20.315584 6081 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:11:20.317843 6203 tablet_service.cc:1467] Processing CreateTablet for tablet f2c92d0e6109427ea596569974e10e51 (DEFAULT_TABLE table=test-table [id=5e3913e5149c4ff79662ed964503f2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:20.318027 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c92d0e6109427ea596569974e10e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:20.319257 6161 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap starting.
I20240501 02:11:20.319588 6161 tablet_bootstrap.cc:654] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:20.320200 6161 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: No bootstrap required, opened a new log
I20240501 02:11:20.320261 6161 ts_tablet_manager.cc:1367] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240501 02:11:20.320278 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.317326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:44671 (local address 127.0.0.1:35454)
0501 02:11:20.317520 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.317528 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.317585 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.317891 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.317892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.317894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:20.318118 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.318119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.319209 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.319210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.320038 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.320041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.320110 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.320115 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.320151 (+ 36us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.320195 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":23,"thread_start_us":111,"threads_started":1}
I20240501 02:11:20.320380 6161 raft_consensus.cc:348] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.320448 6161 raft_consensus.cc:374] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:20.320468 6161 raft_consensus.cc:729] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d6f3e5f16c4b5d8c5b279ea33d3336, State: Initialized, Role: FOLLOWER
I20240501 02:11:20.320520 6161 consensus_queue.cc:260] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.320617 6161 ts_tablet_manager.cc:1398] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:20.321144 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.317606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.22.12.66:44671)
0501 02:11:20.317747 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.317753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.317760 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:11:20.317793 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.317795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.317808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.317881 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:20.318210 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.319138 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.320134 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.320989 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.321032 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.321060 (+ 28us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.321089 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20240501 02:11:20.321740 6341 tablet_service.cc:1467] Processing CreateTablet for tablet f2c92d0e6109427ea596569974e10e51 (DEFAULT_TABLE table=test-table [id=5e3913e5149c4ff79662ed964503f2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:20.321923 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.317326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:40541 (local address 127.0.0.1:54792)
0501 02:11:20.317381 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.317391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.317428 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.319509 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.319510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.319512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:20.319694 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.319696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.320885 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.320886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.321731 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.321734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.321793 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.321797 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.321820 (+ 23us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.321859 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20240501 02:11:20.322067 6341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c92d0e6109427ea596569974e10e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:20.322467 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.317515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.22.12.67:40541)
0501 02:11:20.319326 (+ 1811us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.319333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.319344 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:11:20.319385 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.319387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.319410 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.319507 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:20.319779 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.320818 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.321835 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.322319 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.322358 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.322388 (+ 30us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.322414 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1771,"thread_start_us":80,"threads_started":1}
I20240501 02:11:20.323019 6473 tablet_service.cc:1467] Processing CreateTablet for tablet f2c92d0e6109427ea596569974e10e51 (DEFAULT_TABLE table=test-table [id=5e3913e5149c4ff79662ed964503f2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:20.323298 6473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c92d0e6109427ea596569974e10e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:20.324784 6549 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Bootstrap starting.
I20240501 02:11:20.325261 6549 tablet_bootstrap.cc:654] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:20.325500 6549 log.cc:824] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:20.325613 6550 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Bootstrap starting.
I20240501 02:11:20.326169 6550 tablet_bootstrap.cc:654] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:20.326439 6549 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: No bootstrap required, opened a new log
I20240501 02:11:20.326443 6550 log.cc:824] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:20.326507 6549 ts_tablet_manager.cc:1367] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:11:20.327241 6550 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: No bootstrap required, opened a new log
I20240501 02:11:20.327322 6550 ts_tablet_manager.cc:1367] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:11:20.328621 6549 raft_consensus.cc:348] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.328783 6549 raft_consensus.cc:374] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:20.328809 6549 raft_consensus.cc:729] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3135be291ab34db5801da64c1ece1781, State: Initialized, Role: FOLLOWER
I20240501 02:11:20.328917 6549 consensus_queue.cc:260] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.329164 6549 ts_tablet_manager.cc:1398] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:11:20.329208 6407 heartbeater.cc:503] Master 127.22.12.126:41739 was elected leader, sending a full tablet report...
I20240501 02:11:20.329408 6550 raft_consensus.cc:348] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.329624 6550 raft_consensus.cc:374] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:20.329660 6550 raft_consensus.cc:729] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ed6edd16564293875fef463e3fd9d8, State: Initialized, Role: FOLLOWER
I20240501 02:11:20.329751 6550 consensus_queue.cc:260] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.330026 6550 ts_tablet_manager.cc:1398] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:11:20.330096 6539 heartbeater.cc:503] Master 127.22.12.126:41739 was elected leader, sending a full tablet report...
W20240501 02:11:20.417768 6408 tablet.cc:2369] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:11:20.534999 6270 tablet.cc:2369] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:11:20.542125 6540 tablet.cc:2369] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:11:20.578866 6275 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:20.579005 6275 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.579399 6275 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
I20240501 02:11:20.581492 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.579561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:40541 (local address 127.22.12.65:55119)
0501 02:11:20.579703 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.579724 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.579780 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.579900 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.579900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.579906 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:20.580075 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.580077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.580694 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.580695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.581267 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.581269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.581333 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.581338 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.581365 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.581404 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20240501 02:11:20.581609 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.579546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:44671 (local address 127.22.12.65:48141)
0501 02:11:20.579667 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.579675 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:20.579762 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:20.579895 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:20.579896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:20.579898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:20.580078 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.580079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.580783 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.580784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:20.581338 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:20.581339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.581401 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.581404 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:11:20.581430 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:11:20.581466 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20240501 02:11:20.581969 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.579627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:55119 (local address 127.22.12.67:40541)
0501 02:11:20.579674 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.579676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.579787 (+ 111us) server_negotiation.cc:407] Connection header received
0501 02:11:20.579838 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.579839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.579847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.579889 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:20.580146 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.580624 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.581355 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.581806 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.581816 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.581871 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.581920 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:11:20.581982 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.579613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:48141 (local address 127.22.12.66:44671)
0501 02:11:20.579660 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.579662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.579760 (+ 98us) server_negotiation.cc:407] Connection header received
0501 02:11:20.579824 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.579825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.579835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.579879 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:20.580213 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.580688 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.581417 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.581863 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.581871 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.581894 (+ 23us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.581926 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:11:20.582273 6361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c92d0e6109427ea596569974e10e51" candidate_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3135be291ab34db5801da64c1ece1781" is_pre_election: true
I20240501 02:11:20.582285 6493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c92d0e6109427ea596569974e10e51" candidate_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ed6edd16564293875fef463e3fd9d8" is_pre_election: true
I20240501 02:11:20.582406 6361 raft_consensus.cc:2454] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d6f3e5f16c4b5d8c5b279ea33d3336 in term 0.
I20240501 02:11:20.582406 6493 raft_consensus.cc:2454] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d6f3e5f16c4b5d8c5b279ea33d3336 in term 0.
I20240501 02:11:20.582624 6158 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3135be291ab34db5801da64c1ece1781, c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters:
I20240501 02:11:20.582768 6275 raft_consensus.cc:2781] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:11:20.582832 6275 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:11:20.582847 6275 raft_consensus.cc:3037] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:20.583463 6275 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.583599 6275 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 1 election: Requested vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
I20240501 02:11:20.583761 6493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c92d0e6109427ea596569974e10e51" candidate_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ed6edd16564293875fef463e3fd9d8"
I20240501 02:11:20.583766 6361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c92d0e6109427ea596569974e10e51" candidate_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3135be291ab34db5801da64c1ece1781"
I20240501 02:11:20.583838 6493 raft_consensus.cc:3037] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:20.583839 6361 raft_consensus.cc:3037] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:20.584692 6361 raft_consensus.cc:2454] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d6f3e5f16c4b5d8c5b279ea33d3336 in term 1.
I20240501 02:11:20.584692 6493 raft_consensus.cc:2454] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d6f3e5f16c4b5d8c5b279ea33d3336 in term 1.
I20240501 02:11:20.584885 6158 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3135be291ab34db5801da64c1ece1781, c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters:
I20240501 02:11:20.584992 6275 raft_consensus.cc:2781] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:11:20.585114 6275 raft_consensus.cc:686] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 LEADER]: Becoming Leader. State: Replica: c0d6f3e5f16c4b5d8c5b279ea33d3336, State: Running, Role: LEADER
I20240501 02:11:20.585186 6275 consensus_queue.cc:237] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:20.585924 6080 catalog_manager.cc:5484] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } health_report { overall_health: HEALTHY } } }
I20240501 02:11:20.604763 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.602721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.22.12.65:38483)
0501 02:11:20.602862 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:20.602863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:20.602872 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:11:20.602911 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:20.602912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:20.602929 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:20.603005 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:11:20.603259 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.603732 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:20.604262 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:20.604394 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:20.604435 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:11:20.604525 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:11:20.604607 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:20.604643 (+ 36us) server_negotiation.cc:299] Negotiation successful
0501 02:11:20.604676 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20240501 02:11:20.606168 6361 raft_consensus.cc:1261] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Refusing update from remote peer c0d6f3e5f16c4b5d8c5b279ea33d3336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:11:20.606519 6493 raft_consensus.cc:1261] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Refusing update from remote peer c0d6f3e5f16c4b5d8c5b279ea33d3336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:11:20.606786 6557 consensus_queue.cc:1035] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:11:20.606914 6275 consensus_queue.cc:1035] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:11:20.610697 6562 mvcc.cc:205] Tried to move back new op lower bound from 7022712752560619520 to 7022712752477638656. Current Snapshot: MvccSnapshot[applied={T|T < 7022712752560619520 or (T in {7022712752560619520})}]
I20240501 02:11:20.610817 6563 mvcc.cc:205] Tried to move back new op lower bound from 7022712752560619520 to 7022712752477638656. Current Snapshot: MvccSnapshot[applied={T|T < 7022712752560619520 or (T in {7022712752560619520})}]
I20240501 02:11:20.611047 6080 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49658:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:11:20.611135 6080 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49658:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:11:20.611814 6080 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Sending DeleteTablet for 3 replicas of tablet f2c92d0e6109427ea596569974e10e51
I20240501 02:11:20.612092 6473 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c92d0e6109427ea596569974e10e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:20 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:54792
I20240501 02:11:20.612123 6341 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c92d0e6109427ea596569974e10e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:20 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:35454
I20240501 02:11:20.612177 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6119
I20240501 02:11:20.612740 6571 tablet_replica.cc:330] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: stopping tablet replica
I20240501 02:11:20.612818 6571 raft_consensus.cc:2229] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:20.612928 6571 raft_consensus.cc:2258] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:20.612922 6570 tablet_replica.cc:330] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: stopping tablet replica
I20240501 02:11:20.612989 6570 raft_consensus.cc:2229] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:20.613060 6570 raft_consensus.cc:2258] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:20.613379 6571 ts_tablet_manager.cc:1875] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:20.613399 6570 ts_tablet_manager.cc:1875] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:20.614885 6570 ts_tablet_manager.cc:1888] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:20.614885 6571 ts_tablet_manager.cc:1888] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:20.614957 6570 log.cc:1196] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/f2c92d0e6109427ea596569974e10e51
I20240501 02:11:20.614957 6571 log.cc:1196] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/f2c92d0e6109427ea596569974e10e51
I20240501 02:11:20.615218 6570 ts_tablet_manager.cc:1909] T f2c92d0e6109427ea596569974e10e51 P d9ed6edd16564293875fef463e3fd9d8: Deleting consensus metadata
I20240501 02:11:20.615221 6571 ts_tablet_manager.cc:1909] T f2c92d0e6109427ea596569974e10e51 P 3135be291ab34db5801da64c1ece1781: Deleting consensus metadata
I20240501 02:11:20.615609 6066 catalog_manager.cc:4830] TS d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): tablet f2c92d0e6109427ea596569974e10e51 (table test-table [id=5e3913e5149c4ff79662ed964503f2e0]) successfully deleted
I20240501 02:11:20.615640 6068 catalog_manager.cc:4830] TS 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): tablet f2c92d0e6109427ea596569974e10e51 (table test-table [id=5e3913e5149c4ff79662ed964503f2e0]) successfully deleted
W20240501 02:11:20.617874 6068 connection.cc:535] client connection to 127.22.12.65:38483 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 02:11:20.617965 6068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 02:11:20.618052 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6273
I20240501 02:11:20.618923 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:20.618804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38483 (local address 127.0.0.1:34954)
0501 02:11:20.618852 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:20.618885 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:11:20.619065 6068 catalog_manager.cc:4573] TS c0d6f3e5f16c4b5d8c5b279ea33d3336 (127.22.12.65:38483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2c92d0e6109427ea596569974e10e51: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38483: connect: Connection refused (error 111)
I20240501 02:11:20.622599 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6410
I20240501 02:11:20.626978 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6048
I20240501 02:11:40.632752 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41739
--webserver_interface=127.22.12.126
--webserver_port=40103
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:40.712203 6573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:40.712405 6573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:40.712433 6573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:40.713750 6573 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240501 02:11:40.713793 6573 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240501 02:11:40.713809 6573 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240501 02:11:40.713821 6573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:11:40.713836 6573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:40.713852 6573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:40.713873 6573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:11:40.713897 6573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:11:40.715571 6573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:40.715806 6573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:40.715986 6573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:40.720283 6581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:40.720458 6578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:40.720878 6573 server_base.cc:1034] running on GCE node
W20240501 02:11:40.721179 6579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:40.721490 6573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:40.721735 6573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:40.722862 6573 hybrid_clock.cc:648] HybridClock initialized: now 1714529500722846 us; error 38 us; skew 500 ppm
May 01 02:11:40 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529500, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:40.725282 6573 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:11:40.725687 6573 webserver.cc:460] Webserver started at http://127.22.12.126:40103/ using document root <none> and password file <none>
I20240501 02:11:40.725852 6573 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:40.725886 6573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:40.727133 6573 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:11:40.727645 6588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:40.727824 6573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:11:40.727907 6573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "bd10818eed9f458f81b6ee4230ca066a"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "2bf543d15b56c55f3b010f33daf02ea4"
server_key_iv: "dc06475182bbf62f0bb8bcd6a44c201e"
server_key_version: "encryptionkey@0"
I20240501 02:11:40.728204 6573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:40.743518 6573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:40.745543 6573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:40.745671 6573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:40.749658 6573 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41739
I20240501 02:11:40.749711 6640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41739 every 8 connection(s)
I20240501 02:11:40.750041 6573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:11:40.751438 6641 sys_catalog.cc:263] Verifying existing consensus state
I20240501 02:11:40.751957 6641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap starting.
I20240501 02:11:40.757589 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6573
I20240501 02:11:40.757858 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:38483
--local_ip_for_outbound_sockets=127.22.12.65
--tserver_master_addrs=127.22.12.126:41739
--webserver_port=36673
--webserver_interface=127.22.12.65
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:11:40.761363 6641 log.cc:824] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:40.764279 6641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:11:40.764612 6641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Bootstrap complete.
I20240501 02:11:40.767679 6641 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:40.767966 6641 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd10818eed9f458f81b6ee4230ca066a, State: Initialized, Role: FOLLOWER
I20240501 02:11:40.768126 6641 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:40.768196 6641 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:11:40.768234 6641 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:11:40.768276 6641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 2 FOLLOWER]: Advancing to term 3
I20240501 02:11:40.769516 6641 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:40.769626 6641 leader_election.cc:302] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd10818eed9f458f81b6ee4230ca066a; no voters:
I20240501 02:11:40.769798 6641 leader_election.cc:288] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [CANDIDATE]: Term 3 election: Requested vote from peers
I20240501 02:11:40.769863 6645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 3 FOLLOWER]: Leader election won for term 3
I20240501 02:11:40.770022 6645 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [term 3 LEADER]: Becoming Leader. State: Replica: bd10818eed9f458f81b6ee4230ca066a, State: Running, Role: LEADER
I20240501 02:11:40.770042 6641 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:11:40.770110 6645 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } }
I20240501 02:11:40.770412 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bd10818eed9f458f81b6ee4230ca066a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } } }
I20240501 02:11:40.770498 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:40.770426 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd10818eed9f458f81b6ee4230ca066a. Latest consensus state: current_term: 3 leader_uuid: "bd10818eed9f458f81b6ee4230ca066a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd10818eed9f458f81b6ee4230ca066a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41739 } } }
I20240501 02:11:40.770627 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:40.770889 6651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:11:40.771661 6651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:11:40.771770 6651 catalog_manager.cc:1259] Loaded cluster ID: a8a4435457ab456180c374d8d26baa64
I20240501 02:11:40.771801 6651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:11:40.773018 6651 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:11:40.773201 6651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Loaded TSK: 0
I20240501 02:11:40.773248 6651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd10818eed9f458f81b6ee4230ca066a: Loaded TSK: 1
F20240501 02:11:40.773442 6651 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7feb0547bdcd google::LogMessage::Fail() at ??:0
@ 0x7feb0547fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7feb0547b7cc google::LogMessage::Flush() at ??:0
@ 0x7feb0547cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feb03ccd2f6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7feb03ccabf3 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7feb07c9361e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7feb07c9383a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7feb07c93983 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7feb07c53627 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7feb07c5e463 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7feb06575a08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7feb0656a3a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feb0688d6db start_thread at ??:0
@ 0x7feb049d071f clone at ??:0
W20240501 02:11:40.873072 6643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:40.873437 6643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:40.873476 6643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:40.875634 6643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:40.875708 6643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:40.875749 6643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:11:40.878300 6643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:38483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=36673
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:40.878595 6643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:40.878818 6643 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:40.883484 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:40.883836 6643 server_base.cc:1034] running on GCE node
W20240501 02:11:40.884214 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:40.884390 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:40.884570 6643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:40.884780 6643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:40.885919 6643 hybrid_clock.cc:648] HybridClock initialized: now 1714529500885906 us; error 33 us; skew 500 ppm
May 01 02:11:40 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529500, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:40.894099 6643 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:11:40.894611 6643 webserver.cc:460] Webserver started at http://127.22.12.65:36673/ using document root <none> and password file <none>
I20240501 02:11:40.894809 6643 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:40.894853 6643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:40.896064 6643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:11:40.896692 6692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:40.896883 6643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:11:40.896965 6643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "e744b2014e89b444615fa04b316dda16"
server_key_iv: "3299022a611d7f45a2c3f816ce1b2df3"
server_key_version: "encryptionkey@0"
I20240501 02:11:40.897269 6643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:40.919595 6643 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:40.921660 6643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:40.921778 6643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:40.921981 6643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:40.922411 6699 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 02:11:40.923657 6643 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 02:11:40.923709 6643 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240501 02:11:40.923744 6643 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 02:11:40.924607 6643 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 02:11:40.924640 6643 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 02:11:40.924726 6699 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap starting.
I20240501 02:11:40.931206 6643 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38483
I20240501 02:11:40.931346 6806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38483 every 8 connection(s)
I20240501 02:11:40.931578 6643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:11:40.933562 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6643
I20240501 02:11:40.933843 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:44671
--local_ip_for_outbound_sockets=127.22.12.66
--tserver_master_addrs=127.22.12.126:41739
--webserver_port=46667
--webserver_interface=127.22.12.66
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:11:40.936573 6699 log.cc:824] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:40.939353 6699 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 02:11:40.939653 6699 tablet_bootstrap.cc:492] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Bootstrap complete.
I20240501 02:11:40.940778 6699 ts_tablet_manager.cc:1367] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240501 02:11:40.942025 6699 raft_consensus.cc:348] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:40.942333 6699 raft_consensus.cc:729] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d6f3e5f16c4b5d8c5b279ea33d3336, State: Initialized, Role: FOLLOWER
I20240501 02:11:40.942482 6699 consensus_queue.cc:260] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:40.942798 6699 ts_tablet_manager.cc:1398] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20240501 02:11:40.990633 6809 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:11:40.932252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:43619)
0501 02:11:40.933782 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:40.933819 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:40.934648 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:40.990432 (+ 55784us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1483,"thread_start_us":64,"threads_started":1}
W20240501 02:11:40.990890 6807 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:41739 (0 consecutive failures): Network error: Failed to ping master at 127.22.12.126:41739: Client connection negotiation failed: client connection to 127.22.12.126:41739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 02:11:40.991344 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:40.991211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:34187)
0501 02:11:40.991279 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:40.991319 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20240501 02:11:40.991698 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:40.991607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:59841)
0501 02:11:40.991654 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:40.991679 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240501 02:11:40.991801 6807 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 02:11:41.016429 6810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:41.016630 6810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:41.016654 6810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:41.017998 6810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:41.018043 6810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:41.018067 6810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:11:41.019706 6810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:44671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=46667
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:41.019912 6810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:41.020094 6810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:41.024222 6818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:41.024344 6820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:41.024682 6817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:41.025076 6810 server_base.cc:1034] running on GCE node
I20240501 02:11:41.025266 6810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:41.025471 6810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:41.026574 6810 hybrid_clock.cc:648] HybridClock initialized: now 1714529501026564 us; error 24 us; skew 500 ppm
May 01 02:11:41 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529501, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:41.028903 6810 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:11:41.029326 6810 webserver.cc:460] Webserver started at http://127.22.12.66:46667/ using document root <none> and password file <none>
I20240501 02:11:41.029485 6810 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:41.029521 6810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:41.030746 6810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:41.031296 6827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.031486 6810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.031569 6810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "3135be291ab34db5801da64c1ece1781"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "df58ebe72ed25a0508e1c8ece7fdfd3b"
server_key_iv: "655ed5312d990a33a645945cba206e60"
server_key_version: "encryptionkey@0"
I20240501 02:11:41.031878 6810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:41.049093 6810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:41.051206 6810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:41.051324 6810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:41.051523 6810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:41.051818 6810 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:41.051852 6810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.051882 6810 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:41.051908 6810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.057161 6810 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:44671
I20240501 02:11:41.057211 6940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:44671 every 8 connection(s)
I20240501 02:11:41.057497 6810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:11:41.058426 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.058129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:47701)
0501 02:11:41.058259 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.058315 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
W20240501 02:11:41.058630 6941 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:41739 (0 consecutive failures): Network error: Failed to ping master at 127.22.12.126:41739: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
I20240501 02:11:41.058862 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.058788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:56743)
0501 02:11:41.058825 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.058847 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20240501 02:11:41.059062 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.059004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:38693)
0501 02:11:41.059034 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.059050 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20240501 02:11:41.059137 6941 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 02:11:41.059868 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6810
I20240501 02:11:41.060078 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:40541
--local_ip_for_outbound_sockets=127.22.12.67
--tserver_master_addrs=127.22.12.126:41739
--webserver_port=42679
--webserver_interface=127.22.12.67
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:41.138409 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:41.138643 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:41.138672 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:41.139984 6944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:41.140023 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:41.140048 6944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:11:41.141633 6944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:40541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=42679
--tserver_master_addrs=127.22.12.126:41739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:41.141821 6944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:41.141973 6944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:41.146462 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:41.146596 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:41.146801 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:41.147249 6944 server_base.cc:1034] running on GCE node
I20240501 02:11:41.147426 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:41.147630 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:41.148788 6944 hybrid_clock.cc:648] HybridClock initialized: now 1714529501148768 us; error 31 us; skew 500 ppm
May 01 02:11:41 dist-test-slave-9tkh krb5kdc[5504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529501, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:41.151134 6944 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:11:41.151516 6944 webserver.cc:460] Webserver started at http://127.22.12.67:42679/ using document root <none> and password file <none>
I20240501 02:11:41.151671 6944 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:41.151707 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:41.152848 6944 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:41.153343 6959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.153519 6944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.153589 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "d9ed6edd16564293875fef463e3fd9d8"
format_stamp: "Formatted at 2024-05-01 02:10:59 on dist-test-slave-9tkh"
server_key: "8f6a4ca473b8cd35636c1ff5b5136e2d"
server_key_iv: "20c94ee367e334e375557d828223a0d7"
server_key_version: "encryptionkey@0"
I20240501 02:11:41.153858 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:41.170774 6944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:41.172636 6944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:41.172753 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:41.172933 6944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:41.173235 6944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:41.173269 6944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.173290 6944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:41.173300 6944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:41.178282 6944 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:40541
I20240501 02:11:41.178341 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:40541 every 8 connection(s)
I20240501 02:11:41.178678 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:11:41.179687 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.179340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:43593)
0501 02:11:41.179493 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.179557 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
W20240501 02:11:41.179886 7073 heartbeater.cc:644] Failed to heartbeat to 127.22.12.126:41739 (0 consecutive failures): Network error: Failed to ping master at 127.22.12.126:41739: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
I20240501 02:11:41.180126 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.180049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:49933)
0501 02:11:41.180084 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.180108 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20240501 02:11:41.180372 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.180308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:45111)
0501 02:11:41.180343 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.180360 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20240501 02:11:41.180449 7073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 02:11:41.184279 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 6944
I20240501 02:11:41.992492 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:41.992209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:38145)
0501 02:11:41.992376 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:41.992424 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20240501 02:11:42.059568 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.059372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:60633)
0501 02:11:42.059488 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:42.059515 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20240501 02:11:42.181088 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.180817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:38453)
0501 02:11:42.180975 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:42.181017 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20240501 02:11:42.653664 7080 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:42.653810 7080 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:42.654132 7080 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
May 01 02:11:42 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529500, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.66@KRBTEST.COM
May 01 02:11:42 dist-test-slave-9tkh krb5kdc[5504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529500, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.67@KRBTEST.COM
I20240501 02:11:42.664384 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.654317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:44671 (local address 127.22.12.65:42483)
0501 02:11:42.654520 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:42.654543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:42.654594 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:42.655465 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:42.655468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:42.656017 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:42.656414 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:42.656422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.657926 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:42.657928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:42.658488 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:42.658490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.658643 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:42.659433 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:42.659437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:42.660893 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:42.662803 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:42.662809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:42.662812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:42.663063 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:42.663329 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:42.663330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:42.663397 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:42.663852 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:42.663860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:42.664011 (+ 151us) client_negotiation.cc:769] Sending connection context
0501 02:11:42.664147 (+ 136us) client_negotiation.cc:240] Negotiation successful
0501 02:11:42.664283 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":15,"thread_start_us":52,"threads_started":1}
I20240501 02:11:42.664646 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.654469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:42483 (local address 127.22.12.66:44671)
0501 02:11:42.654610 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:42.654620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:42.654642 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:11:42.655181 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:42.655184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:42.655244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:42.655411 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:42.656528 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.657798 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:42.658782 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.659007 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:42.661006 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:42.661017 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:42.661018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:42.661044 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:42.662727 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663181 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:42.663183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:42.663184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:42.663252 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663453 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:42.663453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:42.663454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:42.663713 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:42.663819 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:42.664245 (+ 426us) server_negotiation.cc:299] Negotiation successful
0501 02:11:42.664377 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20240501 02:11:42.664713 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.654405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:35667 (local address 127.22.12.67:40541)
0501 02:11:42.654589 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:42.654630 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:42.654644 (+ 14us) server_negotiation.cc:407] Connection header received
0501 02:11:42.655180 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:42.655184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:42.655244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:42.655411 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:42.656530 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.657798 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:42.658915 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.659167 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:42.661381 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:42.661392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:42.661392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:42.661424 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:42.663231 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663593 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:42.663595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:42.663596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:42.663653 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663863 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:42.663865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:42.663866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:42.664112 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:42.664195 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:42.664411 (+ 216us) server_negotiation.cc:299] Negotiation successful
0501 02:11:42.664641 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20240501 02:11:42.664752 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.654317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:40541 (local address 127.22.12.65:35667)
0501 02:11:42.654443 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:42.654454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:42.654511 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:42.655465 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:42.655468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:42.656024 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:42.656414 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:42.656422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.657926 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:42.657928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:42.658501 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:42.658503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:42.658644 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:42.659276 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:42.659296 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:42.661271 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:42.663310 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:42.663312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:42.663472 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:42.663739 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:42.663740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:42.663741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:42.663772 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:42.664203 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:42.664204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:42.664254 (+ 50us) client_negotiation.cc:769] Sending connection context
0501 02:11:42.664331 (+ 77us) client_negotiation.cc:240] Negotiation successful
0501 02:11:42.664423 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20240501 02:11:42.665381 6696 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
W20240501 02:11:42.665643 6694 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
I20240501 02:11:42.665696 6694 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters: 3135be291ab34db5801da64c1ece1781, d9ed6edd16564293875fef463e3fd9d8
I20240501 02:11:42.665822 7080 raft_consensus.cc:2726] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:11:42.993155 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:42.993011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:59429)
0501 02:11:42.993063 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:42.993109 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240501 02:11:43.060204 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:43.059987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:59801)
0501 02:11:43.060108 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:43.060160 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20240501 02:11:43.181887 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:43.181595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:56093)
0501 02:11:43.181766 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:43.181827 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20240501 02:11:43.993947 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:43.993701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:47703)
0501 02:11:43.993852 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:43.993895 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20240501 02:11:44.060855 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:44.060621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:43469)
0501 02:11:44.060752 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:44.060796 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20240501 02:11:44.182754 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:44.182484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:44985)
0501 02:11:44.182651 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:44.182698 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20240501 02:11:44.621930 7090 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:44.622035 7090 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:44.622222 7090 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
W20240501 02:11:44.622648 6694 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
W20240501 02:11:44.622761 6696 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
I20240501 02:11:44.622808 6696 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters: 3135be291ab34db5801da64c1ece1781, d9ed6edd16564293875fef463e3fd9d8
I20240501 02:11:44.622879 7090 raft_consensus.cc:2726] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:11:44.994747 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:44.994476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:53223)
0501 02:11:44.994653 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:44.994696 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
I20240501 02:11:45.061420 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:45.061239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:40097)
0501 02:11:45.061344 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:45.061374 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20240501 02:11:45.183540 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:45.183280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:46171)
0501 02:11:45.183439 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:45.183486 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20240501 02:11:45.995347 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:45.995160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:38405)
0501 02:11:45.995272 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:45.995299 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20240501 02:11:46.062072 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:46.061842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:46175)
0501 02:11:46.061981 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:46.062024 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20240501 02:11:46.152467 7096 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:46.152559 7096 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:46.152690 7096 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
W20240501 02:11:46.153110 6696 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
W20240501 02:11:46.153198 6694 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
I20240501 02:11:46.153234 6694 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters: 3135be291ab34db5801da64c1ece1781, d9ed6edd16564293875fef463e3fd9d8
I20240501 02:11:46.153295 7096 raft_consensus.cc:2726] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:11:46.184232 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:46.183991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:49393)
0501 02:11:46.184131 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:46.184168 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20240501 02:11:46.996098 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:46.995821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:51127)
0501 02:11:46.995996 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:46.996043 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20240501 02:11:47.062748 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:47.062505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:34893)
0501 02:11:47.062656 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:47.062699 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20240501 02:11:47.184996 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:47.184732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:50367)
0501 02:11:47.184895 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:47.184941 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20240501 02:11:47.996996 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:47.996703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:44441)
0501 02:11:47.996849 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:47.996895 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20240501 02:11:48.063438 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:48.063186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:44579)
0501 02:11:48.063343 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:48.063388 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20240501 02:11:48.185678 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:48.185463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:42055)
0501 02:11:48.185597 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:48.185630 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20240501 02:11:48.997691 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:48.997472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:35645)
0501 02:11:48.997598 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:48.997641 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20240501 02:11:49.064144 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:49.063914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:37457)
0501 02:11:49.064050 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:49.064095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20240501 02:11:49.186558 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:49.186232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:57155)
0501 02:11:49.186429 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:49.186487 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20240501 02:11:49.667856 7107 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:49.667958 7107 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:49.668123 7107 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
W20240501 02:11:49.668534 6696 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
W20240501 02:11:49.668609 6694 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
I20240501 02:11:49.668649 6694 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters: 3135be291ab34db5801da64c1ece1781, d9ed6edd16564293875fef463e3fd9d8
I20240501 02:11:49.668711 7107 raft_consensus.cc:2726] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:11:49.998358 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:49.998134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:47395)
0501 02:11:49.998271 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:49.998307 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20240501 02:11:50.064850 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:50.064630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:47183)
0501 02:11:50.064760 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:50.064800 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20240501 02:11:50.187429 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:50.187125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:49241)
0501 02:11:50.187342 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:50.187376 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20240501 02:11:50.999111 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:50.998878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:34171)
0501 02:11:50.999028 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:50.999063 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20240501 02:11:51.065553 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:51.065319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:41749)
0501 02:11:51.065460 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:51.065500 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20240501 02:11:51.188282 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:51.187997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:58347)
0501 02:11:51.188154 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:51.188214 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20240501 02:11:51.999771 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:51.999529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:58611)
0501 02:11:51.999678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:51.999721 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20240501 02:11:52.066241 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:52.065987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:56183)
0501 02:11:52.066138 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:52.066181 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20240501 02:11:52.189004 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:52.188781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:59819)
0501 02:11:52.188915 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:52.188953 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20240501 02:11:53.000427 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:53.000213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:35385)
0501 02:11:53.000346 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:53.000379 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20240501 02:11:53.066936 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:53.066722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:42089)
0501 02:11:53.066847 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:53.066886 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20240501 02:11:53.189831 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:53.189547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:48519)
0501 02:11:53.189720 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:53.189760 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20240501 02:11:54.001058 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:54.000849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:57275)
0501 02:11:54.000972 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:54.001011 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20240501 02:11:54.067643 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:54.067414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:60239)
0501 02:11:54.067553 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:54.067593 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20240501 02:11:54.190582 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:54.190315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:58213)
0501 02:11:54.190449 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:54.190493 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20240501 02:11:55.001690 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:55.001475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:57945)
0501 02:11:55.001600 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:55.001639 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20240501 02:11:55.068346 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:55.068137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:49915)
0501 02:11:55.068255 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:55.068296 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20240501 02:11:55.191359 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:55.191083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:44349)
0501 02:11:55.191267 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:55.191308 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20240501 02:11:56.002409 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:56.002175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.65:39741)
0501 02:11:56.002321 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:56.002359 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20240501 02:11:56.069090 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:56.068857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.66:54269)
0501 02:11:56.068988 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:56.069033 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20240501 02:11:56.146843 7128 raft_consensus.cc:482] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:56.146942 7128 raft_consensus.cc:504] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3135be291ab34db5801da64c1ece1781" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 44671 } } peers { permanent_uuid: "d9ed6edd16564293875fef463e3fd9d8" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 40541 } } peers { permanent_uuid: "c0d6f3e5f16c4b5d8c5b279ea33d3336" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38483 } }
I20240501 02:11:56.147110 7128 leader_election.cc:288] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671), d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541)
W20240501 02:11:56.147424 6696 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3135be291ab34db5801da64c1ece1781 (127.22.12.66:44671): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
W20240501 02:11:56.147554 6694 leader_election.cc:341] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9ed6edd16564293875fef463e3fd9d8 (127.22.12.67:40541): Not found: Tablet not found: f2c92d0e6109427ea596569974e10e51
I20240501 02:11:56.147604 6694 leader_election.cc:302] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0d6f3e5f16c4b5d8c5b279ea33d3336; no voters: 3135be291ab34db5801da64c1ece1781, d9ed6edd16564293875fef463e3fd9d8
I20240501 02:11:56.147672 7128 raft_consensus.cc:2726] T f2c92d0e6109427ea596569974e10e51 P c0d6f3e5f16c4b5d8c5b279ea33d3336 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 02:11:56.188665 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6643
I20240501 02:11:56.192086 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:56.191891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:41739 (local address 127.22.12.67:51577)
0501 02:11:56.192004 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:56.192037 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.126:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20240501 02:11:56.193323 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6810
I20240501 02:11:56.197682 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 6944
2024-05-01T02:11:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58110 ms)
[----------] 30 tests from SecurityITest (266792 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:11:56 dist-test-slave-9tkh krb5kdc[7132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:11:56 dist-test-slave-9tkh krb5kdc[7132](info): set up 2 sockets
May 01 02:11:56 dist-test-slave-9tkh krb5kdc[7132](info): commencing operation
krb5kdc: starting...
W20240501 02:11:57.090039 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:11:57.204844 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:33551
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:57.286299 7145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:57.286507 7145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:57.286556 7145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:57.287950 7145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:11:57.288012 7145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:57.288024 7145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:57.288034 7145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:11:57.288046 7145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:11:57.289762 7145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:57.289976 7145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:57.290138 7145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:57.294252 7150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.294607 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.294880 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:57.295495 7145 server_base.cc:1034] running on GCE node
I20240501 02:11:57.295780 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:57.296013 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:57.297145 7145 hybrid_clock.cc:648] HybridClock initialized: now 1714529517297129 us; error 34 us; skew 500 ppm
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:57.299767 7145 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:11:57.300216 7145 webserver.cc:460] Webserver started at http://127.22.12.126:40559/ using document root <none> and password file <none>
I20240501 02:11:57.300400 7145 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:57.300442 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:57.300554 7145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:57.301662 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "1ecf80bfec054c9487622b958f415747"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "ff9a8949855321da55b6c27a3ad756da"
server_key_iv: "8d1a16c69c42bcd012383034749f2858"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.301980 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "1ecf80bfec054c9487622b958f415747"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "ff9a8949855321da55b6c27a3ad756da"
server_key_iv: "8d1a16c69c42bcd012383034749f2858"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.303321 7145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:11:57.304087 7160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.304279 7145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:57.304368 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "1ecf80bfec054c9487622b958f415747"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "ff9a8949855321da55b6c27a3ad756da"
server_key_iv: "8d1a16c69c42bcd012383034749f2858"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.304436 7145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:57.316115 7145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:57.318282 7145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:57.318411 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:57.322214 7145 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:33551
I20240501 02:11:57.322245 7212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:33551 every 8 connection(s)
I20240501 02:11:57.322590 7145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:11:57.323426 7213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:57.325695 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747: Bootstrap starting.
I20240501 02:11:57.326275 7213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:57.326575 7213 log.cc:824] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:57.327250 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747: No bootstrap required, opened a new log
I20240501 02:11:57.328717 7213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecf80bfec054c9487622b958f415747" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33551 } }
I20240501 02:11:57.328852 7213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:57.328869 7213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecf80bfec054c9487622b958f415747, State: Initialized, Role: FOLLOWER
I20240501 02:11:57.329000 7213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecf80bfec054c9487622b958f415747" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33551 } }
I20240501 02:11:57.329066 7213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:11:57.329097 7213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:11:57.329138 7213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:57.329404 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7145
I20240501 02:11:57.329517 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:11:57.329664 22577 external_mini_cluster.cc:1428] Setting key d5b0a363af790bf07f9ce85010fd7cf0
I20240501 02:11:57.329742 7213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecf80bfec054c9487622b958f415747" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33551 } }
I20240501 02:11:57.329850 7213 leader_election.cc:302] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ecf80bfec054c9487622b958f415747; no voters:
I20240501 02:11:57.330039 7213 leader_election.cc:288] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:11:57.330071 7216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:11:57.330379 7213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:11:57.330468 7216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [term 1 LEADER]: Becoming Leader. State: Replica: 1ecf80bfec054c9487622b958f415747, State: Running, Role: LEADER
I20240501 02:11:57.330665 7216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecf80bfec054c9487622b958f415747" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33551 } }
I20240501 02:11:57.331076 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ecf80bfec054c9487622b958f415747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecf80bfec054c9487622b958f415747" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33551 } } }
I20240501 02:11:57.331171 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:57.331388 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ecf80bfec054c9487622b958f415747. Latest consensus state: current_term: 1 leader_uuid: "1ecf80bfec054c9487622b958f415747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecf80bfec054c9487622b958f415747" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 33551 } } }
I20240501 02:11:57.331455 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747 [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:57.331377 7227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:11:57.332165 7227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:11:57.333840 7227 catalog_manager.cc:1347] Generated new cluster ID: 7b7a441bd54847309416e92f7218dc3b
I20240501 02:11:57.333904 7227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:57.339257 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.330116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.22.12.126:33551)
0501 02:11:57.330352 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.330367 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.330396 (+ 29us) server_negotiation.cc:407] Connection header received
0501 02:11:57.330525 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.330559 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.330615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.330785 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:57.331872 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.333054 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.333980 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.334246 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.336092 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:57.336118 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:57.336118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:57.336135 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:57.337802 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.338109 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.338112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.338112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.338162 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.338302 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.338303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.338303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.338511 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:57.338721 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.338837 (+ 116us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.338973 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20240501 02:11:57.364271 7227 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:11:57.364848 7227 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:11:57.383606 7227 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747: Generated new TSK 0
I20240501 02:11:57.383822 7227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:11:57.417757 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33551
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:57.501739 7239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:57.501951 7239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:57.501977 7239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:57.503396 7239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:57.503444 7239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:57.503471 7239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:11:57.505151 7239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:57.505373 7239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:57.505543 7239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:57.509862 7247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.510098 7244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.510280 7245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:57.510668 7239 server_base.cc:1034] running on GCE node
I20240501 02:11:57.510874 7239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:57.511078 7239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:57.512204 7239 hybrid_clock.cc:648] HybridClock initialized: now 1714529517512191 us; error 32 us; skew 500 ppm
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:57.514622 7239 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:11:57.515050 7239 webserver.cc:460] Webserver started at http://127.22.12.65:43775/ using document root <none> and password file <none>
I20240501 02:11:57.515223 7239 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:57.515260 7239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:57.515359 7239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:57.516355 7239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "9833885ffdb2424c9171b3615c1b5545"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "da9f740c0b41191a12f75110bd4d7347"
server_key_iv: "31f37fe2f9839bf60731928a2f805117"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.516635 7239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "9833885ffdb2424c9171b3615c1b5545"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "da9f740c0b41191a12f75110bd4d7347"
server_key_iv: "31f37fe2f9839bf60731928a2f805117"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.517834 7239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:11:57.518620 7254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.518818 7239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:57.518904 7239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "9833885ffdb2424c9171b3615c1b5545"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "da9f740c0b41191a12f75110bd4d7347"
server_key_iv: "31f37fe2f9839bf60731928a2f805117"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.518965 7239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:57.538245 7239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:57.539904 7239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:57.539999 7239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:57.540174 7239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:57.540472 7239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:57.540503 7239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.540525 7239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:57.540547 7239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.545646 7239 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:44551
I20240501 02:11:57.545717 7367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:44551 every 8 connection(s)
I20240501 02:11:57.545979 7239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:57.553069 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7239
I20240501 02:11:57.553206 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:11:57.553391 22577 external_mini_cluster.cc:1428] Setting key f0b55e26216b333038dd7b3a9767596d
I20240501 02:11:57.554610 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.546672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:35551 (local address 127.22.12.126:33551)
0501 02:11:57.546723 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.546725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.546870 (+ 145us) server_negotiation.cc:407] Connection header received
0501 02:11:57.547355 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.547356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.547378 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.547428 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:57.548448 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.548952 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.549511 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.549690 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.551631 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:57.551637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:57.551638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:57.551645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:57.552972 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.553465 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.553466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.553466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.553494 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.553687 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.553688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.553689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.553800 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:57.553851 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.554269 (+ 418us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.554439 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:11:57.554659 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.546641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:33551 (local address 127.22.12.65:35551)
0501 02:11:57.546769 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.546793 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.547252 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.547496 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.547499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.547818 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:57.548341 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.548346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.549035 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.549036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.549433 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.549434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.549571 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.550120 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:57.550138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:57.551523 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:57.553078 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:57.553086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:57.553090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:57.553388 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:57.553555 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:57.553555 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:57.553556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:57.553627 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:57.553860 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:57.553867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:57.554027 (+ 160us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.554235 (+ 208us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.554412 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20240501 02:11:57.555572 7368 heartbeater.cc:347] Connected to a master server at 127.22.12.126:33551
I20240501 02:11:57.555688 7368 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:57.555910 7368 heartbeater.cc:511] Master 127.22.12.126:33551 requested a full tablet report, sending...
I20240501 02:11:57.556385 7177 ts_manager.cc:194] Registered new tserver with Master: 9833885ffdb2424c9171b3615c1b5545 (127.22.12.65:44551)
I20240501 02:11:57.557037 7177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:35551
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:11:57.579766 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33551
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:57.659233 7373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:57.659444 7373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:57.659467 7373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:57.660826 7373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:57.660877 7373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:57.660902 7373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:11:57.662497 7373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:57.662739 7373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:57.662904 7373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:57.667083 7381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.667282 7378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.667613 7379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:57.667954 7373 server_base.cc:1034] running on GCE node
I20240501 02:11:57.668141 7373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:57.668349 7373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:57.669492 7373 hybrid_clock.cc:648] HybridClock initialized: now 1714529517669472 us; error 32 us; skew 500 ppm
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:57.671900 7373 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:11:57.672335 7373 webserver.cc:460] Webserver started at http://127.22.12.66:35575/ using document root <none> and password file <none>
I20240501 02:11:57.672513 7373 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:57.672556 7373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:57.672663 7373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:57.673655 7373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "4d74dcb585d34b70885ebae90f584e10"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "91bb2c71b23206c9cba123c01746e7b7"
server_key_iv: "5be380c06ff45cb8b22d74c7a844ca5d"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.673978 7373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "4d74dcb585d34b70885ebae90f584e10"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "91bb2c71b23206c9cba123c01746e7b7"
server_key_iv: "5be380c06ff45cb8b22d74c7a844ca5d"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.675213 7373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:11:57.675859 7388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.676020 7373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:11:57.676097 7373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "4d74dcb585d34b70885ebae90f584e10"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "91bb2c71b23206c9cba123c01746e7b7"
server_key_iv: "5be380c06ff45cb8b22d74c7a844ca5d"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.676159 7373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:57.693857 7373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:57.695621 7373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:57.695741 7373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:57.695928 7373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:57.696216 7373 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:57.696249 7373 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.696271 7373 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:57.696295 7373 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.701659 7373 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:34011
I20240501 02:11:57.701753 7501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:34011 every 8 connection(s)
I20240501 02:11:57.702000 7373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:11:57.704201 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7373
I20240501 02:11:57.704314 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:11:57.704468 22577 external_mini_cluster.cc:1428] Setting key bb91065b98182ce3e18b09ea3d6ccd9d
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:57.710915 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.702731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:46613 (local address 127.22.12.126:33551)
0501 02:11:57.702804 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.702806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.702965 (+ 159us) server_negotiation.cc:407] Connection header received
0501 02:11:57.703458 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.703460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.703472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.703532 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:57.704615 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.705193 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.705741 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.705902 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.708222 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:57.708227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:57.708227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:57.708236 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:57.709699 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.710065 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.710066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.710067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.710092 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.710265 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.710266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.710267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.710394 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:57.710462 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.710722 (+ 260us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.710821 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 02:11:57.710953 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.702699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:33551 (local address 127.22.12.66:46613)
0501 02:11:57.702855 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.702880 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.703351 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.703601 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.703605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.703909 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:57.704468 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.704473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.705267 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.705268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.705661 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.705663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.705794 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.706419 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:57.706446 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:57.708113 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:57.709787 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:57.709792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:57.709795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:57.709995 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:57.710156 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:57.710157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:57.710157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:57.710218 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:57.710449 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:57.710452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:57.710598 (+ 146us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.710689 (+ 91us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.710801 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20240501 02:11:57.711462 7502 heartbeater.cc:347] Connected to a master server at 127.22.12.126:33551
I20240501 02:11:57.711551 7502 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:57.711763 7502 heartbeater.cc:511] Master 127.22.12.126:33551 requested a full tablet report, sending...
I20240501 02:11:57.712188 7177 ts_manager.cc:194] Registered new tserver with Master: 4d74dcb585d34b70885ebae90f584e10 (127.22.12.66:34011)
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
I20240501 02:11:57.712708 7177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:46613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:11:57.729274 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33551
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:11:57.807750 7507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:57.807965 7507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:57.807988 7507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:57.809533 7507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:57.809607 7507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:57.809648 7507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:11:57.811400 7507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:57.811609 7507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:57.811758 7507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:57.815578 7512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.815917 7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:57.816267 7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:57.816689 7507 server_base.cc:1034] running on GCE node
I20240501 02:11:57.816871 7507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:57.817062 7507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:57.818202 7507 hybrid_clock.cc:648] HybridClock initialized: now 1714529517818188 us; error 28 us; skew 500 ppm
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:11:57.820685 7507 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:11:57.821182 7507 webserver.cc:460] Webserver started at http://127.22.12.67:43979/ using document root <none> and password file <none>
I20240501 02:11:57.821373 7507 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:57.821417 7507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:57.821529 7507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:57.822630 7507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "3506cd6a02104569a47cff750542103f"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "3461b12fcdc8005bb5ad802f6aeda8e6"
server_key_iv: "1a287ba4ac6cece1b57e577e56008a8f"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.822942 7507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "3506cd6a02104569a47cff750542103f"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "3461b12fcdc8005bb5ad802f6aeda8e6"
server_key_iv: "1a287ba4ac6cece1b57e577e56008a8f"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.824139 7507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:11:57.824800 7522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.824997 7507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:11:57.825084 7507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "3506cd6a02104569a47cff750542103f"
format_stamp: "Formatted at 2024-05-01 02:11:57 on dist-test-slave-9tkh"
server_key: "3461b12fcdc8005bb5ad802f6aeda8e6"
server_key_iv: "1a287ba4ac6cece1b57e577e56008a8f"
server_key_version: "encryptionkey@0"
I20240501 02:11:57.825145 7507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:57.841487 7507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:57.843608 7507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:57.843739 7507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:57.843950 7507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:57.844259 7507 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:57.844295 7507 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.844322 7507 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:57.844352 7507 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:57.849960 7507 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:33737
I20240501 02:11:57.849997 7635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:33737 every 8 connection(s)
I20240501 02:11:57.850282 7507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:11:57.853761 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7507
I20240501 02:11:57.853875 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:11:57.854055 22577 external_mini_cluster.cc:1428] Setting key 1e4b9b05e7e22a719f87aa0540c782cc
May 01 02:11:57 dist-test-slave-9tkh krb5kdc[7132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529517, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:11:57.858340 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.850969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:58765 (local address 127.22.12.126:33551)
0501 02:11:57.851014 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.851016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.851179 (+ 163us) server_negotiation.cc:407] Connection header received
0501 02:11:57.851657 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.851658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.851668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.851746 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:57.852803 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.853305 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.853996 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.854135 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.856151 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:57.856155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:57.856156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:57.856162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:57.857456 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.857745 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.857746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.857746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.857762 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.857886 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.857887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.857887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.857970 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:57.858011 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.858208 (+ 197us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.858274 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:11:57.858455 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.850925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:33551 (local address 127.22.12.67:58765)
0501 02:11:57.851084 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.851114 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.851545 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.851815 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.851819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.852116 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:57.852715 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.852721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.853381 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.853383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.853868 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.853870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.854062 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.854685 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:11:57.854712 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:57.856076 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:57.857520 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:57.857522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:57.857525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:57.857695 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:57.857804 (+ 109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:11:57.857804 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:11:57.857804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:11:57.857850 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:11:57.858007 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:57.858009 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:57.858115 (+ 106us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.858194 (+ 79us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.858300 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20240501 02:11:57.858891 7636 heartbeater.cc:347] Connected to a master server at 127.22.12.126:33551
I20240501 02:11:57.858978 7636 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:57.859141 7636 heartbeater.cc:511] Master 127.22.12.126:33551 requested a full tablet report, sending...
I20240501 02:11:57.859483 7177 ts_manager.cc:194] Registered new tserver with Master: 3506cd6a02104569a47cff750542103f (127.22.12.67:33737)
I20240501 02:11:57.859835 7177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:58765
I20240501 02:11:57.865797 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:11:57.872678 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.867106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.22.12.126:33551)
0501 02:11:57.867181 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.867183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.867264 (+ 81us) server_negotiation.cc:407] Connection header received
0501 02:11:57.867322 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.867322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.867332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.867378 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:57.868093 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.868576 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.869124 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.869285 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.870255 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:57.870260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:57.870260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:11:57.870266 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:57.871721 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.872008 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.872009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.872010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.872033 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:11:57.872166 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:11:57.872167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:11:57.872167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:11:57.872300 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:57.872394 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.872502 (+ 108us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.872584 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 02:11:57.874349 7177 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:11:57.875828 7177 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:11:57.880698 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.877662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:34011 (local address 127.0.0.1:37596)
0501 02:11:57.877949 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.877956 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.877991 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.878320 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.878326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.878328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:57.878608 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.878610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.879650 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.879652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.880406 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.880408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.880483 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.880526 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.880577 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.880617 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":54,"threads_started":1}
I20240501 02:11:57.881024 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.877629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33737 (local address 127.0.0.1:45270)
0501 02:11:57.877766 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.877793 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.877849 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.878321 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.878324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.878326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:57.878599 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.878603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.880056 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.880057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.880869 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.880872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.880928 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.880932 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.880951 (+ 19us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.880979 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20240501 02:11:57.881265 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.877967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.22.12.66:34011)
0501 02:11:57.878138 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.878147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.878157 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:11:57.878196 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.878198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.878214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.878694 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:57.878777 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.879570 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.880525 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.881055 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.881111 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.881144 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.881185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20240501 02:11:57.881662 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.877804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.22.12.67:33737)
0501 02:11:57.878148 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.878154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.878163 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:11:57.878193 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.878195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.878208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.878280 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:57.878737 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.879970 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.880964 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.881477 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.881525 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.881555 (+ 30us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.881593 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":88,"threads_started":1}
I20240501 02:11:57.881872 7436 tablet_service.cc:1467] Processing CreateTablet for tablet aaa2632f4f544cd5ac31ec8184416935 (DEFAULT_TABLE table=test-table [id=829aa5ec51f442b3b70a0b6709f3c7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:57.882166 7436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa2632f4f544cd5ac31ec8184416935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:57.882177 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.877873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:44551 (local address 127.0.0.1:52100)
0501 02:11:57.877971 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.877980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.878040 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.878378 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.878379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.878381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:57.878632 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.878633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.881222 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.881223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.881971 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.881973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.882045 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.882049 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.882076 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.882116 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20240501 02:11:57.882205 7570 tablet_service.cc:1467] Processing CreateTablet for tablet aaa2632f4f544cd5ac31ec8184416935 (DEFAULT_TABLE table=test-table [id=829aa5ec51f442b3b70a0b6709f3c7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:57.882450 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa2632f4f544cd5ac31ec8184416935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:57.883024 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.877980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.22.12.65:44551)
0501 02:11:57.878139 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.878148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.878157 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:11:57.878196 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.878198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.878223 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.878295 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:57.878711 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.879973 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.882088 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.882640 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.882684 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.882717 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.882760 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20240501 02:11:57.883618 7302 tablet_service.cc:1467] Processing CreateTablet for tablet aaa2632f4f544cd5ac31ec8184416935 (DEFAULT_TABLE table=test-table [id=829aa5ec51f442b3b70a0b6709f3c7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:57.883919 7302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa2632f4f544cd5ac31ec8184416935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:57.884986 7650 tablet_bootstrap.cc:492] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Bootstrap starting.
I20240501 02:11:57.885390 7651 tablet_bootstrap.cc:492] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Bootstrap starting.
I20240501 02:11:57.885650 7650 tablet_bootstrap.cc:654] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:57.885905 7651 tablet_bootstrap.cc:654] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:57.885951 7650 log.cc:824] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:57.886193 7651 log.cc:824] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:57.886701 7650 tablet_bootstrap.cc:492] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: No bootstrap required, opened a new log
I20240501 02:11:57.886767 7650 ts_tablet_manager.cc:1367] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:11:57.886915 7651 tablet_bootstrap.cc:492] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: No bootstrap required, opened a new log
I20240501 02:11:57.886943 7654 tablet_bootstrap.cc:492] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: Bootstrap starting.
I20240501 02:11:57.886966 7651 ts_tablet_manager.cc:1367] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:11:57.887509 7654 tablet_bootstrap.cc:654] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:57.887775 7654 log.cc:824] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:57.888511 7654 tablet_bootstrap.cc:492] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: No bootstrap required, opened a new log
I20240501 02:11:57.888566 7654 ts_tablet_manager.cc:1367] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:11:57.888610 7651 raft_consensus.cc:348] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.888715 7651 raft_consensus.cc:374] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:57.888729 7651 raft_consensus.cc:729] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3506cd6a02104569a47cff750542103f, State: Initialized, Role: FOLLOWER
I20240501 02:11:57.888810 7651 consensus_queue.cc:260] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.888870 7650 raft_consensus.cc:348] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.889007 7650 raft_consensus.cc:374] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:57.889031 7650 raft_consensus.cc:729] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d74dcb585d34b70885ebae90f584e10, State: Initialized, Role: FOLLOWER
I20240501 02:11:57.889024 7651 ts_tablet_manager.cc:1398] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:11:57.889051 7636 heartbeater.cc:503] Master 127.22.12.126:33551 was elected leader, sending a full tablet report...
I20240501 02:11:57.889125 7650 consensus_queue.cc:260] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.889367 7650 ts_tablet_manager.cc:1398] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:11:57.889400 7502 heartbeater.cc:503] Master 127.22.12.126:33551 was elected leader, sending a full tablet report...
I20240501 02:11:57.890854 7654 raft_consensus.cc:348] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.890967 7654 raft_consensus.cc:374] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:57.890988 7654 raft_consensus.cc:729] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9833885ffdb2424c9171b3615c1b5545, State: Initialized, Role: FOLLOWER
I20240501 02:11:57.891079 7654 consensus_queue.cc:260] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.891297 7654 ts_tablet_manager.cc:1398] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:11:57.891371 7368 heartbeater.cc:503] Master 127.22.12.126:33551 was elected leader, sending a full tablet report...
I20240501 02:11:57.901157 7658 raft_consensus.cc:482] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:57.901290 7658 raft_consensus.cc:504] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.901603 7658 leader_election.cc:288] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3506cd6a02104569a47cff750542103f (127.22.12.67:33737), 4d74dcb585d34b70885ebae90f584e10 (127.22.12.66:34011)
I20240501 02:11:57.903749 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.901760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:34011 (local address 127.22.12.65:32805)
0501 02:11:57.901814 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.901825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.901864 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.902010 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.902012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.902014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:57.902253 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.902255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.903009 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.903010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.903569 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.903570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.903626 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.903629 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.903647 (+ 18us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.903679 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240501 02:11:57.903892 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.901797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:33737 (local address 127.22.12.65:45907)
0501 02:11:57.901955 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.901963 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:57.902027 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:57.902154 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:57.902155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:57.902156 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:57.902382 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.902384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.903045 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.903046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:57.903610 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:57.903612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.903679 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.903683 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:57.903712 (+ 29us) client_negotiation.cc:240] Negotiation successful
0501 02:11:57.903746 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20240501 02:11:57.904218 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.901857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:45907 (local address 127.22.12.67:33737)
0501 02:11:57.901910 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.901912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.902024 (+ 112us) server_negotiation.cc:407] Connection header received
0501 02:11:57.902079 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.902080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.902090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.902148 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:57.902449 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.902954 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.903693 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.904120 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.904129 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.904147 (+ 18us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.904171 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:11:57.904320 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.901849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:32805 (local address 127.22.12.66:34011)
0501 02:11:57.901892 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.901894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.901902 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:11:57.901928 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.901929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.901938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.901994 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:57.902324 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.902938 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.903652 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.904210 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.904217 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.904236 (+ 19us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.904259 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:11:57.904500 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa2632f4f544cd5ac31ec8184416935" candidate_uuid: "9833885ffdb2424c9171b3615c1b5545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3506cd6a02104569a47cff750542103f" is_pre_election: true
I20240501 02:11:57.904584 7456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa2632f4f544cd5ac31ec8184416935" candidate_uuid: "9833885ffdb2424c9171b3615c1b5545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d74dcb585d34b70885ebae90f584e10" is_pre_election: true
I20240501 02:11:57.904646 7590 raft_consensus.cc:2454] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9833885ffdb2424c9171b3615c1b5545 in term 0.
I20240501 02:11:57.904726 7456 raft_consensus.cc:2454] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9833885ffdb2424c9171b3615c1b5545 in term 0.
I20240501 02:11:57.904872 7257 leader_election.cc:302] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3506cd6a02104569a47cff750542103f, 9833885ffdb2424c9171b3615c1b5545; no voters:
I20240501 02:11:57.905042 7658 raft_consensus.cc:2781] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:11:57.905092 7658 raft_consensus.cc:482] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:11:57.905113 7658 raft_consensus.cc:3037] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:57.905823 7658 raft_consensus.cc:504] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.905946 7658 leader_election.cc:288] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [CANDIDATE]: Term 1 election: Requested vote from peers 3506cd6a02104569a47cff750542103f (127.22.12.67:33737), 4d74dcb585d34b70885ebae90f584e10 (127.22.12.66:34011)
I20240501 02:11:57.906080 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa2632f4f544cd5ac31ec8184416935" candidate_uuid: "9833885ffdb2424c9171b3615c1b5545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3506cd6a02104569a47cff750542103f"
I20240501 02:11:57.906118 7456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa2632f4f544cd5ac31ec8184416935" candidate_uuid: "9833885ffdb2424c9171b3615c1b5545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d74dcb585d34b70885ebae90f584e10"
I20240501 02:11:57.906162 7590 raft_consensus.cc:3037] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:57.906189 7456 raft_consensus.cc:3037] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:57.906842 7590 raft_consensus.cc:2454] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9833885ffdb2424c9171b3615c1b5545 in term 1.
I20240501 02:11:57.907007 7456 raft_consensus.cc:2454] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9833885ffdb2424c9171b3615c1b5545 in term 1.
I20240501 02:11:57.907007 7257 leader_election.cc:302] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3506cd6a02104569a47cff750542103f, 9833885ffdb2424c9171b3615c1b5545; no voters:
I20240501 02:11:57.907099 7658 raft_consensus.cc:2781] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:11:57.907260 7658 raft_consensus.cc:686] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 1 LEADER]: Becoming Leader. State: Replica: 9833885ffdb2424c9171b3615c1b5545, State: Running, Role: LEADER
I20240501 02:11:57.907362 7658 consensus_queue.cc:237] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } }
I20240501 02:11:57.908124 7175 catalog_manager.cc:5484] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9833885ffdb2424c9171b3615c1b5545 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "9833885ffdb2424c9171b3615c1b5545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9833885ffdb2424c9171b3615c1b5545" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 44551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 } health_report { overall_health: UNKNOWN } } }
I20240501 02:11:57.917774 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.915797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.22.12.65:44551)
0501 02:11:57.915859 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:57.915861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:57.915869 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:11:57.915905 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:57.915906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:57.915921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:57.915983 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:11:57.916217 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.916687 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:57.917287 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:57.917432 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:57.917470 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:11:57.917556 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:11:57.917624 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:57.917676 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 02:11:57.917699 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 02:11:57.919463 7456 raft_consensus.cc:1261] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 1 FOLLOWER]: Refusing update from remote peer 9833885ffdb2424c9171b3615c1b5545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:11:57.919463 7590 raft_consensus.cc:1261] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 1 FOLLOWER]: Refusing update from remote peer 9833885ffdb2424c9171b3615c1b5545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:11:57.919812 7660 consensus_queue.cc:1035] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3506cd6a02104569a47cff750542103f" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:11:57.919934 7658 consensus_queue.cc:1035] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d74dcb585d34b70885ebae90f584e10" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:11:57.922111 7663 mvcc.cc:205] Tried to move back new op lower bound from 7022712905395449856 to 7022712905349099520. Current Snapshot: MvccSnapshot[applied={T|T < 7022712905395449856}]
I20240501 02:11:57.924170 7175 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:11:57.924270 7175 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:11:57.924940 7175 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1ecf80bfec054c9487622b958f415747: Sending DeleteTablet for 3 replicas of tablet aaa2632f4f544cd5ac31ec8184416935
I20240501 02:11:57.925256 7436 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa2632f4f544cd5ac31ec8184416935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:57 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:37596
I20240501 02:11:57.925267 7302 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa2632f4f544cd5ac31ec8184416935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:57 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:52100
I20240501 02:11:57.925441 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7239
I20240501 02:11:57.925484 7673 tablet_replica.cc:330] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545: stopping tablet replica
I20240501 02:11:57.925475 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa2632f4f544cd5ac31ec8184416935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:57 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:45270
I20240501 02:11:57.925557 7673 raft_consensus.cc:2229] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:11:57.925643 7673 raft_consensus.cc:2258] T aaa2632f4f544cd5ac31ec8184416935 P 9833885ffdb2424c9171b3615c1b5545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:57.926059 7675 tablet_replica.cc:330] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: stopping tablet replica
I20240501 02:11:57.926129 7675 raft_consensus.cc:2229] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:57.926220 7675 raft_consensus.cc:2258] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:57.926362 7674 tablet_replica.cc:330] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: stopping tablet replica
I20240501 02:11:57.926441 7674 raft_consensus.cc:2229] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:57.926551 7674 raft_consensus.cc:2258] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:57.927831 7674 ts_tablet_manager.cc:1875] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:57.928529 7675 ts_tablet_manager.cc:1875] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:57.930037 7675 ts_tablet_manager.cc:1888] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:57.930109 7675 log.cc:1196] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/aaa2632f4f544cd5ac31ec8184416935
I20240501 02:11:57.930377 7675 ts_tablet_manager.cc:1909] T aaa2632f4f544cd5ac31ec8184416935 P 3506cd6a02104569a47cff750542103f: Deleting consensus metadata
I20240501 02:11:57.930461 7674 ts_tablet_manager.cc:1888] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:57.930546 7674 log.cc:1196] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/aaa2632f4f544cd5ac31ec8184416935
I20240501 02:11:57.930790 7674 ts_tablet_manager.cc:1909] T aaa2632f4f544cd5ac31ec8184416935 P 4d74dcb585d34b70885ebae90f584e10: Deleting consensus metadata
I20240501 02:11:57.930864 7163 catalog_manager.cc:4830] TS 3506cd6a02104569a47cff750542103f (127.22.12.67:33737): tablet aaa2632f4f544cd5ac31ec8184416935 (table test-table [id=829aa5ec51f442b3b70a0b6709f3c7ff]) successfully deleted
I20240501 02:11:57.931143 7164 catalog_manager.cc:4830] TS 4d74dcb585d34b70885ebae90f584e10 (127.22.12.66:34011): tablet aaa2632f4f544cd5ac31ec8184416935 (table test-table [id=829aa5ec51f442b3b70a0b6709f3c7ff]) successfully deleted
W20240501 02:11:57.931627 7163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:44551 (error 108)
I20240501 02:11:57.931874 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7373
I20240501 02:11:57.932441 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:57.931957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:44551 (local address 127.0.0.1:52106)
0501 02:11:57.932379 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:57.932408 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
W20240501 02:11:57.933921 7163 catalog_manager.cc:4573] TS 9833885ffdb2424c9171b3615c1b5545 (127.22.12.65:44551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aaa2632f4f544cd5ac31ec8184416935: Network error: Client connection negotiation failed: client connection to 127.22.12.65:44551: connect: Connection refused (error 111)
I20240501 02:11:57.936862 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7507
I20240501 02:11:57.941722 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7145
2024-05-01T02:11:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1743 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2024-05-01T02:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:11:57Z Disabled control of system clock
I20240501 02:11:57.967162 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:34511
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:11:58.044426 7680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:58.044642 7680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:58.044663 7680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:58.046157 7680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:11:58.046218 7680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:58.046231 7680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:58.046243 7680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:11:58.046267 7680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:11:58.047864 7680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:34511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:58.048099 7680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:58.048352 7680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:58.052285 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.052673 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.053026 7688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:58.053426 7680 server_base.cc:1034] running on GCE node
I20240501 02:11:58.053637 7680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:58.053933 7680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:58.055063 7680 hybrid_clock.cc:648] HybridClock initialized: now 1714529518055050 us; error 29 us; skew 500 ppm
I20240501 02:11:58.056187 7680 webserver.cc:460] Webserver started at http://127.22.12.126:40101/ using document root <none> and password file <none>
I20240501 02:11:58.056363 7680 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:58.056407 7680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:58.056519 7680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:58.057503 7680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "2604e1b8f5aa4219875759009d424da0"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "a9329a2ea74a63b56abf2e8255c868f2"
server_key_iv: "f1d22101dcc7f0fa7c39a523b7999d3f"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.057803 7680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "2604e1b8f5aa4219875759009d424da0"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "a9329a2ea74a63b56abf2e8255c868f2"
server_key_iv: "f1d22101dcc7f0fa7c39a523b7999d3f"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.058882 7680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:11:58.059458 7694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.059605 7680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:11:58.059677 7680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "2604e1b8f5aa4219875759009d424da0"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "a9329a2ea74a63b56abf2e8255c868f2"
server_key_iv: "f1d22101dcc7f0fa7c39a523b7999d3f"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.059732 7680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:58.082999 7680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:58.083298 7680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:58.083411 7680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:58.087141 7680 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:34511
I20240501 02:11:58.087175 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:34511 every 8 connection(s)
I20240501 02:11:58.087510 7680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:11:58.088114 7747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:58.090616 7747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0: Bootstrap starting.
I20240501 02:11:58.090972 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7680
I20240501 02:11:58.091080 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:11:58.091245 22577 external_mini_cluster.cc:1428] Setting key 8318b0048d60499f409504a87fe242d8
I20240501 02:11:58.091300 7747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:58.091565 7747 log.cc:824] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:58.092339 7747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0: No bootstrap required, opened a new log
I20240501 02:11:58.093513 7747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2604e1b8f5aa4219875759009d424da0" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34511 } }
I20240501 02:11:58.093636 7747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:58.093652 7747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2604e1b8f5aa4219875759009d424da0, State: Initialized, Role: FOLLOWER
I20240501 02:11:58.093739 7747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2604e1b8f5aa4219875759009d424da0" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34511 } }
I20240501 02:11:58.093799 7747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:11:58.093825 7747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:11:58.093871 7747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:58.094388 7747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2604e1b8f5aa4219875759009d424da0" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34511 } }
I20240501 02:11:58.094481 7747 leader_election.cc:302] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2604e1b8f5aa4219875759009d424da0; no voters:
I20240501 02:11:58.094672 7747 leader_election.cc:288] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:11:58.094913 7747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:11:58.095005 7753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:11:58.095185 7753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [term 1 LEADER]: Becoming Leader. State: Replica: 2604e1b8f5aa4219875759009d424da0, State: Running, Role: LEADER
I20240501 02:11:58.095288 7753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2604e1b8f5aa4219875759009d424da0" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34511 } }
I20240501 02:11:58.095619 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2604e1b8f5aa4219875759009d424da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2604e1b8f5aa4219875759009d424da0" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34511 } } }
I20240501 02:11:58.095698 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:58.095831 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2604e1b8f5aa4219875759009d424da0. Latest consensus state: current_term: 1 leader_uuid: "2604e1b8f5aa4219875759009d424da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2604e1b8f5aa4219875759009d424da0" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 34511 } } }
I20240501 02:11:58.095892 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0 [sys.catalog]: This master's current role is: LEADER
I20240501 02:11:58.096015 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.091732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.22.12.126:34511)
0501 02:11:58.091944 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.091952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.091970 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:11:58.092444 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.092452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.092504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.092636 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:58.093027 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.093830 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.094409 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.094662 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.095227 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:58.095262 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:58.095262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:11:58.095434 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:58.095469 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:11:58.095478 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:58.095526 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.095663 (+ 137us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.095722 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20240501 02:11:58.096165 7761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:11:58.096621 7761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:11:58.098008 7761 catalog_manager.cc:1347] Generated new cluster ID: bd5cbbeb04304d31b7c5c418a7dca8e9
I20240501 02:11:58.098071 7761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:11:58.115860 7761 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:11:58.116492 7761 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:11:58.128947 7761 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0: Generated new TSK 0
I20240501 02:11:58.129161 7761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:11:58.139717 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34511
--builtin_ntp_servers=127.22.12.84:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:11:58.219113 7771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:58.219292 7771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:58.219314 7771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:58.220680 7771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:58.220726 7771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:58.220752 7771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:11:58.222170 7771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:58.222376 7771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:58.222661 7771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:58.227285 7777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.227589 7776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.227960 7779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:58.228354 7771 server_base.cc:1034] running on GCE node
I20240501 02:11:58.228533 7771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:58.228744 7771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:58.229887 7771 hybrid_clock.cc:648] HybridClock initialized: now 1714529518229865 us; error 38 us; skew 500 ppm
I20240501 02:11:58.231309 7771 webserver.cc:460] Webserver started at http://127.22.12.65:37451/ using document root <none> and password file <none>
I20240501 02:11:58.231505 7771 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:58.231545 7771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:58.231652 7771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:58.232633 7771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "b21618d7e3dc4e9dae1d9f2641fb266f"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "2bd6cce06a10ba9529304186bf903fa7"
server_key_iv: "cf393794b2f66c8e39597c9f18ba336a"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.232925 7771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "b21618d7e3dc4e9dae1d9f2641fb266f"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "2bd6cce06a10ba9529304186bf903fa7"
server_key_iv: "cf393794b2f66c8e39597c9f18ba336a"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.234050 7771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:11:58.234768 7785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.234915 7771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:11:58.234984 7771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "b21618d7e3dc4e9dae1d9f2641fb266f"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "2bd6cce06a10ba9529304186bf903fa7"
server_key_iv: "cf393794b2f66c8e39597c9f18ba336a"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.235047 7771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:58.256480 7771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:58.256765 7771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:58.256871 7771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:58.257062 7771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:58.257383 7771 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:58.257414 7771 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.257442 7771 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:58.257470 7771 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.262709 7771 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:38933
I20240501 02:11:58.262773 7898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:38933 every 8 connection(s)
I20240501 02:11:58.263054 7771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:11:58.264242 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7771
I20240501 02:11:58.264338 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:11:58.264549 22577 external_mini_cluster.cc:1428] Setting key 01fce6ca403a90bf031a6bac95ba158d
I20240501 02:11:58.265444 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34511
--builtin_ntp_servers=127.22.12.84:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 02:11:58.267740 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.263687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:58955 (local address 127.22.12.126:34511)
0501 02:11:58.263739 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.263741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.263944 (+ 203us) server_negotiation.cc:407] Connection header received
0501 02:11:58.264464 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.264466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.264479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.264582 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:58.265087 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.265675 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.266188 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.266349 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.267489 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:58.267494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:58.267494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:11:58.267498 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:58.267514 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:11:58.267518 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:58.267561 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.267649 (+ 88us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.267681 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:11:58.267905 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.263663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:34511 (local address 127.22.12.65:58955)
0501 02:11:58.263851 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.263878 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.264322 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.264643 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.264648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.264659 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:58.265005 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.265009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.265758 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.265760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.266117 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.266119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.266253 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.267346 (+ 1093us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:11:58.267347 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:58.267375 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:11:58.267399 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:58.267575 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:58.267578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:58.267581 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.267653 (+ 72us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.267729 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20240501 02:11:58.268357 7899 heartbeater.cc:347] Connected to a master server at 127.22.12.126:34511
I20240501 02:11:58.268469 7899 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:58.268669 7899 heartbeater.cc:511] Master 127.22.12.126:34511 requested a full tablet report, sending...
I20240501 02:11:58.269181 7711 ts_manager.cc:194] Registered new tserver with Master: b21618d7e3dc4e9dae1d9f2641fb266f (127.22.12.65:38933)
I20240501 02:11:58.269815 7711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.65:58955
W20240501 02:11:58.344883 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:58.345080 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:58.345111 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:58.346427 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:58.346473 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:58.346506 7902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:11:58.348001 7902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:58.348238 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:58.348471 7902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:58.352672 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.352882 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.353289 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:58.354636 7902 server_base.cc:1034] running on GCE node
I20240501 02:11:58.354857 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:58.355075 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:58.356215 7902 hybrid_clock.cc:648] HybridClock initialized: now 1714529518356202 us; error 35 us; skew 500 ppm
I20240501 02:11:58.357460 7902 webserver.cc:460] Webserver started at http://127.22.12.66:37869/ using document root <none> and password file <none>
I20240501 02:11:58.357640 7902 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:58.357682 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:58.357805 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:58.358825 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "fba693b1a27f4ca9892844c6f2eb28bb"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "a6cdb4c72acb86bb0bb622f6fd21ce82"
server_key_iv: "bbb559bec657c959b03469ca493b136d"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.359098 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "fba693b1a27f4ca9892844c6f2eb28bb"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "a6cdb4c72acb86bb0bb622f6fd21ce82"
server_key_iv: "bbb559bec657c959b03469ca493b136d"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.360203 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:58.360917 7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.361083 7902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:11:58.361157 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "fba693b1a27f4ca9892844c6f2eb28bb"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "a6cdb4c72acb86bb0bb622f6fd21ce82"
server_key_iv: "bbb559bec657c959b03469ca493b136d"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.361212 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:58.381038 7902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:58.381318 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:58.381428 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:58.381641 7902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:58.381954 7902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:58.381986 7902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.382015 7902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:58.382032 7902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.387073 7902 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:40153
I20240501 02:11:58.387138 8029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:40153 every 8 connection(s)
I20240501 02:11:58.387410 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:11:58.390403 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 7902
I20240501 02:11:58.390497 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:11:58.390698 22577 external_mini_cluster.cc:1428] Setting key 8ce79eed00e1ac91219c08dcd70be4a8
I20240501 02:11:58.391782 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34511
--builtin_ntp_servers=127.22.12.84:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 02:11:58.391925 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.388074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:57081 (local address 127.22.12.126:34511)
0501 02:11:58.388138 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.388140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.388339 (+ 199us) server_negotiation.cc:407] Connection header received
0501 02:11:58.388847 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.388849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.388861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.388917 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:58.389376 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.389852 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.390372 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.390554 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.391629 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:58.391632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:58.391632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:11:58.391635 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:58.391651 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:11:58.391654 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:58.391694 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.391807 (+ 113us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.391838 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:11:58.392058 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.388053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:34511 (local address 127.22.12.66:57081)
0501 02:11:58.388224 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.388250 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.388741 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.388982 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.388985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.388994 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:58.389290 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.389293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.389930 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.389931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.390295 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.390296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.390423 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.391484 (+ 1061us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:11:58.391484 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:58.391514 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:11:58.391537 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:58.391721 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:58.391726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:58.391729 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.391812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.391896 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20240501 02:11:58.392477 8030 heartbeater.cc:347] Connected to a master server at 127.22.12.126:34511
I20240501 02:11:58.392565 8030 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:58.392880 8030 heartbeater.cc:511] Master 127.22.12.126:34511 requested a full tablet report, sending...
I20240501 02:11:58.393266 7711 ts_manager.cc:194] Registered new tserver with Master: fba693b1a27f4ca9892844c6f2eb28bb (127.22.12.66:40153)
I20240501 02:11:58.393728 7711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.66:57081
W20240501 02:11:58.469655 8033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:11:58.469834 8033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:11:58.469864 8033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:11:58.471190 8033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:11:58.471235 8033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:11:58.471264 8033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:11:58.472692 8033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:11:58.472903 8033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:11:58.473131 8033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:11:58.477651 8038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.477761 8041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:11:58.478070 8039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:11:58.478430 8033 server_base.cc:1034] running on GCE node
I20240501 02:11:58.478610 8033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:11:58.478811 8033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:11:58.479938 8033 hybrid_clock.cc:648] HybridClock initialized: now 1714529518479925 us; error 29 us; skew 500 ppm
I20240501 02:11:58.481227 8033 webserver.cc:460] Webserver started at http://127.22.12.67:43735/ using document root <none> and password file <none>
I20240501 02:11:58.481408 8033 fs_manager.cc:359] Metadata directory not provided
I20240501 02:11:58.481454 8033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:11:58.481563 8033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:11:58.483284 8033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "77711f6a69d54a51a672e6d9e52705db"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "4617abed9f4a2706814a99cc3b92c3a9"
server_key_iv: "d4c6840800f86e50083466015057f171"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.483997 8033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "77711f6a69d54a51a672e6d9e52705db"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "4617abed9f4a2706814a99cc3b92c3a9"
server_key_iv: "d4c6840800f86e50083466015057f171"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.485670 8033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:11:58.486479 8047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.486646 8033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:11:58.486744 8033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "77711f6a69d54a51a672e6d9e52705db"
format_stamp: "Formatted at 2024-05-01 02:11:58 on dist-test-slave-9tkh"
server_key: "4617abed9f4a2706814a99cc3b92c3a9"
server_key_iv: "d4c6840800f86e50083466015057f171"
server_key_version: "encryptionkey@0"
I20240501 02:11:58.486816 8033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:11:58.542811 8033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:11:58.543128 8033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:11:58.543237 8033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:11:58.543483 8033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:11:58.543843 8033 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:11:58.543887 8033 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.543920 8033 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:11:58.543936 8033 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:11:58.550592 8033 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:43497
I20240501 02:11:58.550642 8160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:43497 every 8 connection(s)
I20240501 02:11:58.550992 8033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:11:58.556023 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.551716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:51957 (local address 127.22.12.126:34511)
0501 02:11:58.551782 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.551784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.552026 (+ 242us) server_negotiation.cc:407] Connection header received
0501 02:11:58.552677 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.552678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.552688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.552733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:58.553325 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.553788 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.554554 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.554668 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.555779 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:58.555783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:58.555784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:11:58.555788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:58.555801 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:11:58.555804 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:58.555845 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.555935 (+ 90us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.555961 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:11:58.556200 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.551678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:34511 (local address 127.22.12.67:51957)
0501 02:11:58.551910 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.551941 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.552554 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.552796 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.552800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.552811 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:11:58.553252 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.553256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.553869 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.553870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.554464 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.554466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.554650 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.555632 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:11:58.555633 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:11:58.555660 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:11:58.555685 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:11:58.555861 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:11:58.555864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:11:58.555867 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.555940 (+ 73us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.556014 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20240501 02:11:58.556700 8161 heartbeater.cc:347] Connected to a master server at 127.22.12.126:34511
I20240501 02:11:58.556788 8161 heartbeater.cc:464] Registering TS with master...
I20240501 02:11:58.556979 8161 heartbeater.cc:511] Master 127.22.12.126:34511 requested a full tablet report, sending...
I20240501 02:11:58.557363 7711 ts_manager.cc:194] Registered new tserver with Master: 77711f6a69d54a51a672e6d9e52705db (127.22.12.67:43497)
I20240501 02:11:58.557394 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8033
I20240501 02:11:58.557482 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:11:58.557662 22577 external_mini_cluster.cc:1428] Setting key 6c3d81c7b5600d2cab60b3e611b8e983
I20240501 02:11:58.557837 7711 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.67:51957
I20240501 02:11:58.558488 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:11:58.562395 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.559760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.22.12.126:34511)
0501 02:11:58.559795 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.559797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.559929 (+ 132us) server_negotiation.cc:407] Connection header received
0501 02:11:58.559957 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.559958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.559970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.560032 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:11:58.560293 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.561050 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.561591 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.561757 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.562152 (+ 395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:11:58.562156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:11:58.562157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:11:58.562162 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:11:58.562175 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:11:58.562179 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:11:58.562239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.562287 (+ 48us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.562312 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:11:58.564203 7710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:11:58.565675 7710 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:11:58.571513 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.567582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:40153 (local address 127.0.0.1:35144)
0501 02:11:58.567718 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.567743 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.567802 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.568477 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.568480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.568482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:58.568687 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.568689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.570205 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.570207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.571190 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.571194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.571275 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.571320 (+ 45us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.571376 (+ 56us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.571421 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20240501 02:11:58.571913 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.568161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.22.12.66:40153)
0501 02:11:58.568304 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.568312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.568323 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:11:58.568364 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.568366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.568385 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.568464 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:58.568759 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.570117 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.571273 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.571659 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.571767 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.571800 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.571839 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20240501 02:11:58.572445 7964 tablet_service.cc:1467] Processing CreateTablet for tablet 05d209c784bb4229b5084d143bfce80c (DEFAULT_TABLE table=test-table [id=94bf894dcff7417696a172cf59776652]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:58.572716 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d209c784bb4229b5084d143bfce80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:58.575294 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.567835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.22.12.65:38933)
0501 02:11:58.568021 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.568029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.568373 (+ 344us) server_negotiation.cc:407] Connection header received
0501 02:11:58.568427 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.568430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.568448 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.568536 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:58.568936 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.570120 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.574481 (+ 4361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.575048 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.575136 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.575173 (+ 37us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.575213 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20240501 02:11:58.575641 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.567802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:43497 (local address 127.0.0.1:45578)
0501 02:11:58.571611 (+ 3809us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.571620 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.571663 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.571891 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.571892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.571894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:58.572064 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.572065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.574605 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.574606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.575450 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.575452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.575501 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.575504 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.575525 (+ 21us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.575557 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3798,"thread_start_us":62,"threads_started":1}
I20240501 02:11:58.575784 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.567630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38933 (local address 127.0.0.1:35436)
0501 02:11:58.568292 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.568302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.568367 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.568635 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.568636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.568639 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:58.568856 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.568857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.573430 (+ 4573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.573431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.574378 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.574381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.574456 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.574460 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.574491 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.575729 (+ 1238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":74,"threads_started":1}
I20240501 02:11:58.575835 8175 tablet_bootstrap.cc:492] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Bootstrap starting.
I20240501 02:11:58.576443 8175 tablet_bootstrap.cc:654] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:58.576370 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 05d209c784bb4229b5084d143bfce80c (DEFAULT_TABLE table=test-table [id=94bf894dcff7417696a172cf59776652]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:58.576660 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d209c784bb4229b5084d143bfce80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:58.576714 8175 log.cc:824] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:58.576768 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.568031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.22.12.67:43497)
0501 02:11:58.568202 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.568220 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.571724 (+ 3504us) server_negotiation.cc:407] Connection header received
0501 02:11:58.571775 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.571778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.571798 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.572129 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:58.572198 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.573344 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.575976 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.576517 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.576615 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.576649 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.576687 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20240501 02:11:58.577349 8095 tablet_service.cc:1467] Processing CreateTablet for tablet 05d209c784bb4229b5084d143bfce80c (DEFAULT_TABLE table=test-table [id=94bf894dcff7417696a172cf59776652]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:11:58.577608 8095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d209c784bb4229b5084d143bfce80c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:11:58.577633 8175 tablet_bootstrap.cc:492] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: No bootstrap required, opened a new log
I20240501 02:11:58.577692 8175 ts_tablet_manager.cc:1367] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:11:58.579892 8175 raft_consensus.cc:348] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.580035 8175 raft_consensus.cc:374] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:58.580060 8175 raft_consensus.cc:729] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba693b1a27f4ca9892844c6f2eb28bb, State: Initialized, Role: FOLLOWER
I20240501 02:11:58.580145 8175 consensus_queue.cc:260] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.580366 8175 ts_tablet_manager.cc:1398] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:11:58.580525 8030 heartbeater.cc:503] Master 127.22.12.126:34511 was elected leader, sending a full tablet report...
I20240501 02:11:58.580976 8177 tablet_bootstrap.cc:492] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Bootstrap starting.
I20240501 02:11:58.581532 8177 tablet_bootstrap.cc:654] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:58.581792 8177 log.cc:824] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:58.582087 8179 tablet_bootstrap.cc:492] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Bootstrap starting.
I20240501 02:11:58.582463 8177 tablet_bootstrap.cc:492] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: No bootstrap required, opened a new log
I20240501 02:11:58.582572 8177 ts_tablet_manager.cc:1367] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:11:58.582653 8179 tablet_bootstrap.cc:654] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Neither blocks nor log segments found. Creating new log.
I20240501 02:11:58.582895 8179 log.cc:824] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Log is configured to *not* fsync() on all Append() calls
I20240501 02:11:58.583479 8179 tablet_bootstrap.cc:492] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: No bootstrap required, opened a new log
I20240501 02:11:58.583532 8179 ts_tablet_manager.cc:1367] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:11:58.584257 8177 raft_consensus.cc:348] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.584393 8177 raft_consensus.cc:374] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:58.584416 8177 raft_consensus.cc:729] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21618d7e3dc4e9dae1d9f2641fb266f, State: Initialized, Role: FOLLOWER
I20240501 02:11:58.584525 8177 consensus_queue.cc:260] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.584784 8177 ts_tablet_manager.cc:1398] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:11:58.584857 7899 heartbeater.cc:503] Master 127.22.12.126:34511 was elected leader, sending a full tablet report...
I20240501 02:11:58.585386 8179 raft_consensus.cc:348] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.585492 8179 raft_consensus.cc:374] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:11:58.585508 8179 raft_consensus.cc:729] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77711f6a69d54a51a672e6d9e52705db, State: Initialized, Role: FOLLOWER
I20240501 02:11:58.585595 8179 consensus_queue.cc:260] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.585793 8179 ts_tablet_manager.cc:1398] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:11:58.585837 8161 heartbeater.cc:503] Master 127.22.12.126:34511 was elected leader, sending a full tablet report...
W20240501 02:11:58.637936 8031 tablet.cc:2369] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:11:58.763692 7900 tablet.cc:2369] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:11:58.801565 8162 tablet.cc:2369] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:11:58.923820 8183 raft_consensus.cc:482] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:11:58.923946 8183 raft_consensus.cc:504] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.924279 8183 leader_election.cc:288] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fba693b1a27f4ca9892844c6f2eb28bb (127.22.12.66:40153), b21618d7e3dc4e9dae1d9f2641fb266f (127.22.12.65:38933)
I20240501 02:11:58.926482 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.924464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38933 (local address 127.22.12.67:58781)
0501 02:11:58.924527 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.924553 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.924599 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.924766 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.924767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.924769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:58.924967 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.924969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.925706 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.925708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.926297 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.926299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.926356 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.926360 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.926380 (+ 20us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.926409 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20240501 02:11:58.926642 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.924479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:40153 (local address 127.22.12.67:46647)
0501 02:11:58.924683 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:58.924694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:11:58.924753 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:11:58.924872 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:11:58.924873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:11:58.924874 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:11:58.925100 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.925102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.925791 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.925792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:11:58.926384 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:11:58.926386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.926467 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.926471 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:11:58.926500 (+ 29us) client_negotiation.cc:240] Negotiation successful
0501 02:11:58.926558 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":144,"threads_started":1}
I20240501 02:11:58.927021 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.924526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:46647 (local address 127.22.12.66:40153)
0501 02:11:58.924590 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.924591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.924765 (+ 174us) server_negotiation.cc:407] Connection header received
0501 02:11:58.924796 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.924797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.924807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.924862 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:58.925194 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.925685 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.926473 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.926924 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.926932 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.926952 (+ 20us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.926974 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 02:11:58.927227 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:58.924567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:58781 (local address 127.22.12.65:38933)
0501 02:11:58.924616 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:58.924618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:58.924651 (+ 33us) server_negotiation.cc:407] Connection header received
0501 02:11:58.924680 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:58.924681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:58.924691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:58.924748 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:11:58.925041 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.925573 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:58.926397 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:58.927098 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:58.927107 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:58.927134 (+ 27us) server_negotiation.cc:299] Negotiation successful
0501 02:11:58.927163 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:11:58.927295 7984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d209c784bb4229b5084d143bfce80c" candidate_uuid: "77711f6a69d54a51a672e6d9e52705db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" is_pre_election: true
I20240501 02:11:58.927457 7984 raft_consensus.cc:2454] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77711f6a69d54a51a672e6d9e52705db in term 0.
I20240501 02:11:58.927484 7853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d209c784bb4229b5084d143bfce80c" candidate_uuid: "77711f6a69d54a51a672e6d9e52705db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" is_pre_election: true
I20240501 02:11:58.927604 7853 raft_consensus.cc:2454] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77711f6a69d54a51a672e6d9e52705db in term 0.
I20240501 02:11:58.927700 8048 leader_election.cc:302] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77711f6a69d54a51a672e6d9e52705db, fba693b1a27f4ca9892844c6f2eb28bb; no voters:
I20240501 02:11:58.927827 8183 raft_consensus.cc:2781] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:11:58.927897 8183 raft_consensus.cc:482] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:11:58.927915 8183 raft_consensus.cc:3037] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:58.928550 8183 raft_consensus.cc:504] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.928678 8183 leader_election.cc:288] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [CANDIDATE]: Term 1 election: Requested vote from peers fba693b1a27f4ca9892844c6f2eb28bb (127.22.12.66:40153), b21618d7e3dc4e9dae1d9f2641fb266f (127.22.12.65:38933)
I20240501 02:11:58.928854 7984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d209c784bb4229b5084d143bfce80c" candidate_uuid: "77711f6a69d54a51a672e6d9e52705db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba693b1a27f4ca9892844c6f2eb28bb"
I20240501 02:11:58.928931 7984 raft_consensus.cc:3037] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:58.929574 7984 raft_consensus.cc:2454] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77711f6a69d54a51a672e6d9e52705db in term 1.
I20240501 02:11:58.929778 8048 leader_election.cc:302] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77711f6a69d54a51a672e6d9e52705db, fba693b1a27f4ca9892844c6f2eb28bb; no voters:
I20240501 02:11:58.929787 7853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d209c784bb4229b5084d143bfce80c" candidate_uuid: "77711f6a69d54a51a672e6d9e52705db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f"
I20240501 02:11:58.929872 8183 raft_consensus.cc:2781] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:11:58.929860 7853 raft_consensus.cc:3037] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:11:58.930035 8183 raft_consensus.cc:686] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 1 LEADER]: Becoming Leader. State: Replica: 77711f6a69d54a51a672e6d9e52705db, State: Running, Role: LEADER
I20240501 02:11:58.930138 8183 consensus_queue.cc:237] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } }
I20240501 02:11:58.930732 7853 raft_consensus.cc:2454] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77711f6a69d54a51a672e6d9e52705db in term 1.
I20240501 02:11:58.931237 7710 catalog_manager.cc:5484] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db reported cstate change: term changed from 0 to 1, leader changed from <none> to 77711f6a69d54a51a672e6d9e52705db (127.22.12.67). New cstate: current_term: 1 leader_uuid: "77711f6a69d54a51a672e6d9e52705db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77711f6a69d54a51a672e6d9e52705db" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 43497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20240501 02:11:59.014350 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:59.012435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.22.12.67:43497)
0501 02:11:59.012477 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:11:59.012479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:11:59.012486 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:11:59.012525 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:11:59.012526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:11:59.012542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:11:59.012605 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:11:59.012822 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:59.013281 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:11:59.013840 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:11:59.013970 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:11:59.014010 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:11:59.014119 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:11:59.014181 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 02:11:59.014255 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 02:11:59.014277 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240501 02:11:59.016044 7853 raft_consensus.cc:1261] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 1 FOLLOWER]: Refusing update from remote peer 77711f6a69d54a51a672e6d9e52705db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:11:59.016043 7984 raft_consensus.cc:1261] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 1 FOLLOWER]: Refusing update from remote peer 77711f6a69d54a51a672e6d9e52705db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:11:59.016391 8183 consensus_queue.cc:1035] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba693b1a27f4ca9892844c6f2eb28bb" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:11:59.016546 8185 consensus_queue.cc:1035] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b21618d7e3dc4e9dae1d9f2641fb266f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:11:59.018575 8188 mvcc.cc:205] Tried to move back new op lower bound from 7022712909887115264 to 7022712909538455552. Current Snapshot: MvccSnapshot[applied={T|T < 7022712909887115264}]
I20240501 02:11:59.018908 8189 mvcc.cc:205] Tried to move back new op lower bound from 7022712909887115264 to 7022712909538455552. Current Snapshot: MvccSnapshot[applied={T|T < 7022712909887115264}]
I20240501 02:11:59.019520 8190 mvcc.cc:205] Tried to move back new op lower bound from 7022712909887115264 to 7022712909538455552. Current Snapshot: MvccSnapshot[applied={T|T < 7022712909887115264}]
I20240501 02:11:59.021613 7711 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43630:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:11:59.021718 7711 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43630:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:11:59.022401 7711 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2604e1b8f5aa4219875759009d424da0: Sending DeleteTablet for 3 replicas of tablet 05d209c784bb4229b5084d143bfce80c
I20240501 02:11:59.022729 8095 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d209c784bb4229b5084d143bfce80c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:59 UTC) from {username='slave'} at 127.0.0.1:45578
I20240501 02:11:59.022729 7833 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d209c784bb4229b5084d143bfce80c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:59 UTC) from {username='slave'} at 127.0.0.1:35436
I20240501 02:11:59.023051 8198 tablet_replica.cc:330] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: stopping tablet replica
I20240501 02:11:59.023052 7964 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d209c784bb4229b5084d143bfce80c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:11:59 UTC) from {username='slave'} at 127.0.0.1:35144
I20240501 02:11:59.023139 8198 raft_consensus.cc:2229] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:59.023237 8198 raft_consensus.cc:2258] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:59.023341 8200 tablet_replica.cc:330] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: stopping tablet replica
I20240501 02:11:59.023411 8200 raft_consensus.cc:2229] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:11:59.023490 8200 raft_consensus.cc:2258] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:59.023559 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7771
I20240501 02:11:59.023680 8198 ts_tablet_manager.cc:1875] T 05d209c784bb4229b5084d143bfce80c P b21618d7e3dc4e9dae1d9f2641fb266f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:59.024076 8199 tablet_replica.cc:330] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: stopping tablet replica
I20240501 02:11:59.024369 8199 raft_consensus.cc:2229] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:11:59.024485 8199 raft_consensus.cc:2258] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:11:59.024514 8200 ts_tablet_manager.cc:1875] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:59.024866 8199 ts_tablet_manager.cc:1875] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:11:59.026297 8200 ts_tablet_manager.cc:1888] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:59.026305 8199 ts_tablet_manager.cc:1888] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:11:59.026357 8199 log.cc:1196] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/05d209c784bb4229b5084d143bfce80c
I20240501 02:11:59.026357 8200 log.cc:1196] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/05d209c784bb4229b5084d143bfce80c
I20240501 02:11:59.026618 8199 ts_tablet_manager.cc:1909] T 05d209c784bb4229b5084d143bfce80c P 77711f6a69d54a51a672e6d9e52705db: Deleting consensus metadata
I20240501 02:11:59.026619 8200 ts_tablet_manager.cc:1909] T 05d209c784bb4229b5084d143bfce80c P fba693b1a27f4ca9892844c6f2eb28bb: Deleting consensus metadata
I20240501 02:11:59.026968 7695 catalog_manager.cc:4830] TS fba693b1a27f4ca9892844c6f2eb28bb (127.22.12.66:40153): tablet 05d209c784bb4229b5084d143bfce80c (table test-table [id=94bf894dcff7417696a172cf59776652]) successfully deleted
I20240501 02:11:59.027037 7695 catalog_manager.cc:4830] TS 77711f6a69d54a51a672e6d9e52705db (127.22.12.67:43497): tablet 05d209c784bb4229b5084d143bfce80c (table test-table [id=94bf894dcff7417696a172cf59776652]) successfully deleted
W20240501 02:11:59.031232 7696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:38933 (error 108)
I20240501 02:11:59.031482 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7902
I20240501 02:11:59.032080 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:11:59.031563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:38933 (local address 127.0.0.1:35450)
0501 02:11:59.032014 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0501 02:11:59.032048 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432}
W20240501 02:11:59.032557 7696 catalog_manager.cc:4573] TS b21618d7e3dc4e9dae1d9f2641fb266f (127.22.12.65:38933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05d209c784bb4229b5084d143bfce80c: Network error: Client connection negotiation failed: client connection to 127.22.12.65:38933: connect: Connection refused (error 111)
I20240501 02:11:59.038566 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8033
I20240501 02:11:59.043613 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 7680
2024-05-01T02:11:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1101 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:11:59 dist-test-slave-9tkh krb5kdc[8204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:11:59 dist-test-slave-9tkh krb5kdc[8204](info): set up 2 sockets
May 01 02:11:59 dist-test-slave-9tkh krb5kdc[8204](info): commencing operation
krb5kdc: starting...
W20240501 02:11:59.934234 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:00.045701 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:32963
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:32963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:00.123490 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:00.123709 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:00.123742 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:00.125129 8217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:00.125181 8217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:00.125200 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:00.125214 8217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:00.125228 8217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:00.127003 8217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:32963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:32963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:00.127230 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:00.127418 8217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:00.131351 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.131590 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.132011 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:00.132454 8217 server_base.cc:1034] running on GCE node
I20240501 02:12:00.132737 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:00.133000 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:00.134124 8217 hybrid_clock.cc:648] HybridClock initialized: now 1714529520134117 us; error 39 us; skew 500 ppm
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:00.136484 8217 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:00.136916 8217 webserver.cc:460] Webserver started at http://127.22.12.126:45415/ using document root <none> and password file <none>
I20240501 02:12:00.137081 8217 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:00.137123 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:00.137238 8217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:00.138295 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "d2ec58052e474fa0a25c5af260d8a6e2"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "c5c32f994e30ac128234b862a6a579b3"
server_key_iv: "27896030326617e3f87065c35503460f"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.138659 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "d2ec58052e474fa0a25c5af260d8a6e2"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "c5c32f994e30ac128234b862a6a579b3"
server_key_iv: "27896030326617e3f87065c35503460f"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.139878 8217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:00.140558 8232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.140769 8217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:00.140872 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "d2ec58052e474fa0a25c5af260d8a6e2"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "c5c32f994e30ac128234b862a6a579b3"
server_key_iv: "27896030326617e3f87065c35503460f"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.140931 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:00.147768 8217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:00.149293 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:00.149389 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:00.152923 8217 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:32963
I20240501 02:12:00.152964 8284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:32963 every 8 connection(s)
I20240501 02:12:00.153254 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:00.154064 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:00.156253 8285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2: Bootstrap starting.
I20240501 02:12:00.156800 8285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:00.157045 8285 log.cc:824] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:00.157655 8285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2: No bootstrap required, opened a new log
I20240501 02:12:00.159106 8285 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32963 } }
I20240501 02:12:00.159229 8285 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:00.159243 8285 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ec58052e474fa0a25c5af260d8a6e2, State: Initialized, Role: FOLLOWER
I20240501 02:12:00.159369 8285 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32963 } }
I20240501 02:12:00.159425 8285 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:00.159446 8285 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:00.159487 8285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:00.160005 8285 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32963 } }
I20240501 02:12:00.160090 8285 leader_election.cc:302] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ec58052e474fa0a25c5af260d8a6e2; no voters:
I20240501 02:12:00.160117 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8217
I20240501 02:12:00.160203 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:00.160240 8285 leader_election.cc:288] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:00.160288 8288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:00.160368 22577 external_mini_cluster.cc:1428] Setting key efe905b3641a8638a81e92488c8f5399
I20240501 02:12:00.160521 8285 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:00.160727 8288 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [term 1 LEADER]: Becoming Leader. State: Replica: d2ec58052e474fa0a25c5af260d8a6e2, State: Running, Role: LEADER
I20240501 02:12:00.160895 8288 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32963 } }
I20240501 02:12:00.161262 8289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32963 } } }
I20240501 02:12:00.161346 8289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:00.161491 8290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ec58052e474fa0a25c5af260d8a6e2. Latest consensus state: current_term: 1 leader_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ec58052e474fa0a25c5af260d8a6e2" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 32963 } } }
I20240501 02:12:00.161533 8290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:00.161929 8299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:00.162400 8299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:00.163901 8299 catalog_manager.cc:1347] Generated new cluster ID: 48a55d465ef34c95aa5f618d031638e4
I20240501 02:12:00.163959 8299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:00.168975 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.160825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.22.12.126:32963)
0501 02:12:00.160966 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.160978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.161051 (+ 73us) server_negotiation.cc:407] Connection header received
0501 02:12:00.161219 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.161226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.161280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.161972 (+ 692us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:00.162401 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.163322 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.164179 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.164391 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.166146 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:00.166167 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:00.166167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:00.166183 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:00.167750 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.168052 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.168055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.168055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.168103 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.168242 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.168243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.168243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.168428 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:00.168548 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.168663 (+ 115us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.168795 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20240501 02:12:00.194994 8299 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:00.195808 8299 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:00.201105 8299 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2: Generated new TSK 0
I20240501 02:12:00.201332 8299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:12:00.229643 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:32963
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:00.310364 8311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:00.310590 8311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:00.310631 8311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:00.312026 8311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:00.312079 8311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:00.312104 8311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:12:00.313817 8311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:00.314034 8311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:00.314234 8311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:00.318660 8319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.318722 8317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.319083 8316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:00.319485 8311 server_base.cc:1034] running on GCE node
I20240501 02:12:00.319689 8311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:00.319912 8311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:00.321054 8311 hybrid_clock.cc:648] HybridClock initialized: now 1714529520321037 us; error 35 us; skew 500 ppm
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:00.323778 8311 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:12:00.324188 8311 webserver.cc:460] Webserver started at http://127.22.12.65:40221/ using document root <none> and password file <none>
I20240501 02:12:00.324364 8311 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:00.324402 8311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:00.324506 8311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:00.325485 8311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "2c8ded37b4784621abda5ddb50b4badd"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "8e9a532b497e02651289b304ef2e1535"
server_key_iv: "4e002f123742812783492e9a08afeb8b"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.325775 8311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "2c8ded37b4784621abda5ddb50b4badd"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "8e9a532b497e02651289b304ef2e1535"
server_key_iv: "4e002f123742812783492e9a08afeb8b"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.326973 8311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20240501 02:12:00.327601 8326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.327795 8311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:00.327879 8311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "2c8ded37b4784621abda5ddb50b4badd"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "8e9a532b497e02651289b304ef2e1535"
server_key_iv: "4e002f123742812783492e9a08afeb8b"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.327940 8311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:00.346711 8311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:00.348323 8311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:00.348439 8311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:00.348647 8311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:00.348932 8311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:00.348964 8311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.348992 8311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:00.349020 8311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.354269 8311 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:33859
I20240501 02:12:00.354326 8439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:33859 every 8 connection(s)
I20240501 02:12:00.354636 8311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:00.363101 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.355308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:60181 (local address 127.22.12.126:32963)
0501 02:12:00.355357 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.355359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.355553 (+ 194us) server_negotiation.cc:407] Connection header received
0501 02:12:00.356028 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.356029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.356040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.356131 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:00.357200 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.357721 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.358255 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.358390 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.360449 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:00.360453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:00.360453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:00.360460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:00.361893 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.362293 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.362294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.362294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.362321 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.362485 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.362485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.362485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.362636 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:00.362686 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.362947 (+ 261us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.363023 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:12:00.363225 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.355277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:32963 (local address 127.22.12.65:60181)
0501 02:12:00.355443 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.355474 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.355938 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.356189 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.356193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.356499 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:00.357076 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.357080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.357804 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.357804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.358173 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.358175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.358270 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.358917 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:00.358935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:00.360365 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:00.361987 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:00.361994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:00.361997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:00.362209 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:00.362382 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:00.362382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:00.362382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:00.362439 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:00.362697 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:00.362703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:00.362824 (+ 121us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.362939 (+ 115us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.363062 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20240501 02:12:00.363703 8440 heartbeater.cc:347] Connected to a master server at 127.22.12.126:32963
I20240501 02:12:00.363787 8440 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:00.363970 8440 heartbeater.cc:511] Master 127.22.12.126:32963 requested a full tablet report, sending...
I20240501 02:12:00.364491 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8311
I20240501 02:12:00.364492 8249 ts_manager.cc:194] Registered new tserver with Master: 2c8ded37b4784621abda5ddb50b4badd (127.22.12.65:33859)
I20240501 02:12:00.364636 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:12:00.364812 22577 external_mini_cluster.cc:1428] Setting key a4b079016354284f38a3992ec5043f1f
I20240501 02:12:00.365342 8249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:60181
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:12:00.389878 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:32963
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:00.466961 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:00.467147 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:00.467170 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:00.468461 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:00.468504 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:00.468528 8445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:12:00.470062 8445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:00.470254 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:00.470404 8445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:00.474476 8453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.474568 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.474951 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:00.475344 8445 server_base.cc:1034] running on GCE node
I20240501 02:12:00.475508 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:00.475694 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:00.476817 8445 hybrid_clock.cc:648] HybridClock initialized: now 1714529520476804 us; error 35 us; skew 500 ppm
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:00.479246 8445 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:12:00.479669 8445 webserver.cc:460] Webserver started at http://127.22.12.66:38101/ using document root <none> and password file <none>
I20240501 02:12:00.479843 8445 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:00.479883 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:00.479990 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:00.480971 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "49c2d2f41d7d414381f0a005279f80d3"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "9ed70edf05c878f5b054419efb4ad0b5"
server_key_iv: "11cca0ab60945c553f22a560ca12404d"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.481266 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "49c2d2f41d7d414381f0a005279f80d3"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "9ed70edf05c878f5b054419efb4ad0b5"
server_key_iv: "11cca0ab60945c553f22a560ca12404d"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.482372 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:00.483095 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.483275 8445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.483350 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "49c2d2f41d7d414381f0a005279f80d3"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "9ed70edf05c878f5b054419efb4ad0b5"
server_key_iv: "11cca0ab60945c553f22a560ca12404d"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.483407 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:00.499696 8445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:00.501587 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:00.501704 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:00.501919 8445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:00.502223 8445 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:00.502256 8445 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.502285 8445 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:00.502311 8445 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.507460 8445 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:40619
I20240501 02:12:00.507521 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:40619 every 8 connection(s)
I20240501 02:12:00.507783 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:00.514180 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8445
I20240501 02:12:00.514286 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:12:00.514448 22577 external_mini_cluster.cc:1428] Setting key b4fd24f52fe252df9a7e6bb4d160fa9f
I20240501 02:12:00.515810 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.508461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:60279 (local address 127.22.12.126:32963)
0501 02:12:00.508513 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.508515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.508674 (+ 159us) server_negotiation.cc:407] Connection header received
0501 02:12:00.509119 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.509120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.509128 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.509189 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:00.510154 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.510673 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.511204 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.511312 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.513179 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:00.513183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:00.513183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:00.513189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:00.514605 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.515021 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.515022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.515023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.515052 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.515234 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.515234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.515235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.515323 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:00.515372 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.515672 (+ 300us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.515745 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:12:00.515972 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.508423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:32963 (local address 127.22.12.66:60279)
0501 02:12:00.508568 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.508596 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.509033 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.509245 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.509248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.509541 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:00.510051 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.510056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.510758 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.510759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.511144 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.511145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.511253 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.511744 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:00.511758 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:00.513084 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:00.514692 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:00.514697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:00.514700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:00.514937 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:00.515119 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:00.515119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:00.515120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:00.515178 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:00.515381 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:00.515386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:00.515533 (+ 147us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.515649 (+ 116us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.515785 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20240501 02:12:00.516438 8574 heartbeater.cc:347] Connected to a master server at 127.22.12.126:32963
I20240501 02:12:00.516534 8574 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:00.516753 8574 heartbeater.cc:511] Master 127.22.12.126:32963 requested a full tablet report, sending...
I20240501 02:12:00.517153 8249 ts_manager.cc:194] Registered new tserver with Master: 49c2d2f41d7d414381f0a005279f80d3 (127.22.12.66:40619)
I20240501 02:12:00.517719 8249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:60279
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:12:00.537532 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:32963
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:00.614398 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:00.614624 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:00.614650 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:00.615934 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:00.615971 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:00.615995 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:12:00.617483 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:00.617661 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:00.617821 8579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:00.621910 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.622087 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.622440 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:00.622876 8579 server_base.cc:1034] running on GCE node
I20240501 02:12:00.623067 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:00.623282 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:00.624428 8579 hybrid_clock.cc:648] HybridClock initialized: now 1714529520624410 us; error 33 us; skew 500 ppm
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:00.626879 8579 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:12:00.627300 8579 webserver.cc:460] Webserver started at http://127.22.12.67:41133/ using document root <none> and password file <none>
I20240501 02:12:00.627481 8579 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:00.627521 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:00.627632 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:00.628598 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "924c79a77c6f442c8af73cfa8ff6db77"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "f3c133b3c7ae0ad0af9889dc30f33b71"
server_key_iv: "2ce110c253e7a1a05bbff09e4b90476c"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.628882 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "924c79a77c6f442c8af73cfa8ff6db77"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "f3c133b3c7ae0ad0af9889dc30f33b71"
server_key_iv: "2ce110c253e7a1a05bbff09e4b90476c"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.629992 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:12:00.630733 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.630931 8579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:00.631059 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "924c79a77c6f442c8af73cfa8ff6db77"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "f3c133b3c7ae0ad0af9889dc30f33b71"
server_key_iv: "2ce110c253e7a1a05bbff09e4b90476c"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.631136 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:00.642037 8579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:00.643815 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:00.643914 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:00.644110 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:00.644398 8579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:00.644450 8579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.644477 8579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:00.644488 8579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.649487 8579 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:36579
I20240501 02:12:00.649538 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:36579 every 8 connection(s)
I20240501 02:12:00.649806 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:12:00.651923 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8579
I20240501 02:12:00.652020 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:12:00.652202 22577 external_mini_cluster.cc:1428] Setting key d9eb1999ed8420fa85b2a3f61ad9115b
May 01 02:12:00 dist-test-slave-9tkh krb5kdc[8204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529520, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:00.657764 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.650472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:40073 (local address 127.22.12.126:32963)
0501 02:12:00.650522 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.650524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.650746 (+ 222us) server_negotiation.cc:407] Connection header received
0501 02:12:00.651157 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.651158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.651168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.651234 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:00.652222 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.652878 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.653441 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.653556 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.655427 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:00.655430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:00.655431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:00.655437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:00.656701 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.657003 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.657004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.657004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.657024 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.657214 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.657215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.657215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.657322 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:00.657385 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.657626 (+ 241us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.657699 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:12:00.657800 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.650450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:32963 (local address 127.22.12.67:40073)
0501 02:12:00.650594 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.650624 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.651053 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.651283 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.651286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.651570 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:00.652126 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.652130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.652959 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.652960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.653378 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.653380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.653485 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.653981 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:00.653994 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:00.655343 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:00.656765 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:00.656768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:00.656770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:00.656945 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:00.657073 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:00.657073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:00.657074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:00.657136 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:00.657378 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:00.657380 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:00.657480 (+ 100us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.657559 (+ 79us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.657659 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":86,"threads_started":1}
I20240501 02:12:00.658309 8708 heartbeater.cc:347] Connected to a master server at 127.22.12.126:32963
I20240501 02:12:00.658394 8708 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:00.658596 8708 heartbeater.cc:511] Master 127.22.12.126:32963 requested a full tablet report, sending...
I20240501 02:12:00.658991 8249 ts_manager.cc:194] Registered new tserver with Master: 924c79a77c6f442c8af73cfa8ff6db77 (127.22.12.67:36579)
I20240501 02:12:00.659363 8249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:40073
I20240501 02:12:00.663904 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:12:00.670601 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.665240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.22.12.126:32963)
0501 02:12:00.665284 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.665285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.665332 (+ 47us) server_negotiation.cc:407] Connection header received
0501 02:12:00.665387 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.665387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.665402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.665443 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:00.666072 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.666562 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.667151 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.667265 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.668241 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:00.668244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:00.668244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:00.668249 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:00.669464 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.669826 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.669827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.669827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.669847 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:00.670073 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:00.670074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:00.670074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:00.670182 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:00.670284 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.670436 (+ 152us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.670486 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 02:12:00.672268 8249 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:12:00.673624 8249 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:12:00.678668 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.675543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:40619 (local address 127.0.0.1:56120)
0501 02:12:00.675981 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.675991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.676056 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.676240 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.676242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.676245 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:00.676572 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.676574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.677521 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.677522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.678339 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.678341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.678409 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.678452 (+ 43us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.678502 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.678586 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":69,"threads_started":1}
I20240501 02:12:00.678839 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.675511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:33859 (local address 127.0.0.1:53158)
0501 02:12:00.675682 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.675708 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.675774 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.676254 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.676255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.676257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:00.676516 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.676518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.677927 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.677928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.678638 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.678642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.678698 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.678703 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.678726 (+ 23us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.678765 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":58,"threads_started":1}
I20240501 02:12:00.679581 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.675816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.22.12.65:33859)
0501 02:12:00.676063 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.676077 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.676084 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:00.676125 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.676127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.676144 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.676599 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:00.676685 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.677822 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.679060 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.679425 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.679467 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.679493 (+ 26us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.679521 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20240501 02:12:00.679653 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.675664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.22.12.66:40619)
0501 02:12:00.675888 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.675895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.676060 (+ 165us) server_negotiation.cc:407] Connection header received
0501 02:12:00.676138 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.676141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.676156 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.676223 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:00.676720 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.677442 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.678905 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.679449 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.679497 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.679531 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.679571 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20240501 02:12:00.679649 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.675816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.22.12.67:36579)
0501 02:12:00.675988 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.675994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.676092 (+ 98us) server_negotiation.cc:407] Connection header received
0501 02:12:00.676137 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.676139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.676154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.676230 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:00.676594 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.677769 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.678900 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.679458 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.679497 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.679531 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.679570 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20240501 02:12:00.679811 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.675699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:36579 (local address 127.0.0.1:40192)
0501 02:12:00.675981 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.675991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.676056 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.676307 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.676308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.676310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:00.676521 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.676522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.677856 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.677857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.678716 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.678719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.678779 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.678782 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.678809 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.678848 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":43,"threads_started":1}
I20240501 02:12:00.680122 8374 tablet_service.cc:1467] Processing CreateTablet for tablet bc3cee417b8d483487008c57474467ee (DEFAULT_TABLE table=test-table [id=e0f9def632054ad0950394fd35ea429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:00.680392 8374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cee417b8d483487008c57474467ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:00.682821 8722 tablet_bootstrap.cc:492] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: Bootstrap starting.
I20240501 02:12:00.683001 8508 tablet_service.cc:1467] Processing CreateTablet for tablet bc3cee417b8d483487008c57474467ee (DEFAULT_TABLE table=test-table [id=e0f9def632054ad0950394fd35ea429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:00.683269 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cee417b8d483487008c57474467ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:00.683437 8722 tablet_bootstrap.cc:654] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:00.683691 8722 log.cc:824] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:00.684334 8722 tablet_bootstrap.cc:492] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: No bootstrap required, opened a new log
I20240501 02:12:00.684386 8722 ts_tablet_manager.cc:1367] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:00.685868 8724 tablet_bootstrap.cc:492] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Bootstrap starting.
I20240501 02:12:00.686172 8642 tablet_service.cc:1467] Processing CreateTablet for tablet bc3cee417b8d483487008c57474467ee (DEFAULT_TABLE table=test-table [id=e0f9def632054ad0950394fd35ea429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:00.686370 8722 raft_consensus.cc:348] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.686472 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cee417b8d483487008c57474467ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:00.686548 8724 tablet_bootstrap.cc:654] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:00.686519 8722 raft_consensus.cc:374] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:00.686584 8722 raft_consensus.cc:729] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8ded37b4784621abda5ddb50b4badd, State: Initialized, Role: FOLLOWER
I20240501 02:12:00.686731 8722 consensus_queue.cc:260] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.686878 8724 log.cc:824] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:00.686992 8722 ts_tablet_manager.cc:1398] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 02:12:00.687132 8440 heartbeater.cc:503] Master 127.22.12.126:32963 was elected leader, sending a full tablet report...
I20240501 02:12:00.687592 8724 tablet_bootstrap.cc:492] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: No bootstrap required, opened a new log
I20240501 02:12:00.687665 8724 ts_tablet_manager.cc:1367] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:00.689186 8727 tablet_bootstrap.cc:492] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Bootstrap starting.
I20240501 02:12:00.689504 8724 raft_consensus.cc:348] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.689644 8724 raft_consensus.cc:374] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:00.689680 8724 raft_consensus.cc:729] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c2d2f41d7d414381f0a005279f80d3, State: Initialized, Role: FOLLOWER
I20240501 02:12:00.689729 8727 tablet_bootstrap.cc:654] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:00.689778 8724 consensus_queue.cc:260] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.690006 8727 log.cc:824] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:00.690035 8724 ts_tablet_manager.cc:1398] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 02:12:00.690057 8574 heartbeater.cc:503] Master 127.22.12.126:32963 was elected leader, sending a full tablet report...
I20240501 02:12:00.690718 8727 tablet_bootstrap.cc:492] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: No bootstrap required, opened a new log
I20240501 02:12:00.690770 8727 ts_tablet_manager.cc:1367] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:00.692322 8727 raft_consensus.cc:348] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.692432 8727 raft_consensus.cc:374] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:00.692447 8727 raft_consensus.cc:729] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924c79a77c6f442c8af73cfa8ff6db77, State: Initialized, Role: FOLLOWER
I20240501 02:12:00.692530 8727 consensus_queue.cc:260] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.692737 8727 ts_tablet_manager.cc:1398] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:12:00.692790 8708 heartbeater.cc:503] Master 127.22.12.126:32963 was elected leader, sending a full tablet report...
I20240501 02:12:00.715082 8725 raft_consensus.cc:482] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:00.715221 8725 raft_consensus.cc:504] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.715502 8725 leader_election.cc:288] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 924c79a77c6f442c8af73cfa8ff6db77 (127.22.12.67:36579), 49c2d2f41d7d414381f0a005279f80d3 (127.22.12.66:40619)
I20240501 02:12:00.718374 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.715660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:36579 (local address 127.22.12.65:39927)
0501 02:12:00.715762 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.715774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.715840 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.716001 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.716002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.716003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:00.716239 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.716241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.717202 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.717203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.718119 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.718121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.718204 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.718205 (+ 1us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.718236 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.718271 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20240501 02:12:00.718385 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.715680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:40619 (local address 127.22.12.65:49887)
0501 02:12:00.715870 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.715879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:00.715932 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:00.716049 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:00.716049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:00.716051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:00.716268 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.716269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.717221 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.717222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:00.718119 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:00.718121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.718198 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.718203 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:12:00.718236 (+ 33us) client_negotiation.cc:240] Negotiation successful
0501 02:12:00.718271 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":11,"thread_start_us":109,"threads_started":1}
I20240501 02:12:00.718842 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.715890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:49887 (local address 127.22.12.66:40619)
0501 02:12:00.715945 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.715947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.715955 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:00.715979 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.715980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.715990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.716039 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:00.716340 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.717114 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.718206 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.718707 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.718712 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.718734 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.718773 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:12:00.718842 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.715748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:39927 (local address 127.22.12.67:36579)
0501 02:12:00.715808 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.715810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.715821 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:12:00.715903 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.715905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.715924 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.715979 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:00.716340 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.717129 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.718204 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.718702 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.718712 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.718734 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.718765 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 02:12:00.719163 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cee417b8d483487008c57474467ee" candidate_uuid: "2c8ded37b4784621abda5ddb50b4badd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924c79a77c6f442c8af73cfa8ff6db77" is_pre_election: true
I20240501 02:12:00.719163 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cee417b8d483487008c57474467ee" candidate_uuid: "2c8ded37b4784621abda5ddb50b4badd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c2d2f41d7d414381f0a005279f80d3" is_pre_election: true
I20240501 02:12:00.719316 8528 raft_consensus.cc:2454] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c8ded37b4784621abda5ddb50b4badd in term 0.
I20240501 02:12:00.719316 8662 raft_consensus.cc:2454] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c8ded37b4784621abda5ddb50b4badd in term 0.
I20240501 02:12:00.719552 8329 leader_election.cc:302] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c8ded37b4784621abda5ddb50b4badd, 49c2d2f41d7d414381f0a005279f80d3; no voters:
I20240501 02:12:00.719679 8725 raft_consensus.cc:2781] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:00.719765 8725 raft_consensus.cc:482] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:00.719797 8725 raft_consensus.cc:3037] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:00.720479 8725 raft_consensus.cc:504] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.720618 8725 leader_election.cc:288] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [CANDIDATE]: Term 1 election: Requested vote from peers 924c79a77c6f442c8af73cfa8ff6db77 (127.22.12.67:36579), 49c2d2f41d7d414381f0a005279f80d3 (127.22.12.66:40619)
I20240501 02:12:00.720763 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cee417b8d483487008c57474467ee" candidate_uuid: "2c8ded37b4784621abda5ddb50b4badd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c2d2f41d7d414381f0a005279f80d3"
I20240501 02:12:00.720772 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cee417b8d483487008c57474467ee" candidate_uuid: "2c8ded37b4784621abda5ddb50b4badd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924c79a77c6f442c8af73cfa8ff6db77"
I20240501 02:12:00.720837 8528 raft_consensus.cc:3037] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:00.720868 8662 raft_consensus.cc:3037] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:00.721516 8662 raft_consensus.cc:2454] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c8ded37b4784621abda5ddb50b4badd in term 1.
I20240501 02:12:00.721518 8528 raft_consensus.cc:2454] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c8ded37b4784621abda5ddb50b4badd in term 1.
I20240501 02:12:00.721711 8329 leader_election.cc:302] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c8ded37b4784621abda5ddb50b4badd, 49c2d2f41d7d414381f0a005279f80d3; no voters:
I20240501 02:12:00.721832 8725 raft_consensus.cc:2781] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:00.721995 8725 raft_consensus.cc:686] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 1 LEADER]: Becoming Leader. State: Replica: 2c8ded37b4784621abda5ddb50b4badd, State: Running, Role: LEADER
I20240501 02:12:00.722097 8725 consensus_queue.cc:237] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } }
I20240501 02:12:00.722968 8249 catalog_manager.cc:5484] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c8ded37b4784621abda5ddb50b4badd (127.22.12.65). New cstate: current_term: 1 leader_uuid: "2c8ded37b4784621abda5ddb50b4badd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c8ded37b4784621abda5ddb50b4badd" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 33859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 } health_report { overall_health: UNKNOWN } } }
I20240501 02:12:00.739370 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.737184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.22.12.65:33859)
0501 02:12:00.737247 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.737248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.737256 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:00.737301 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.737302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.737318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.737383 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:00.737606 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.738101 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.738721 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.738876 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.739046 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:00.739136 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:00.739200 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.739274 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.739283 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:12:00.741137 8662 raft_consensus.cc:1261] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 1 FOLLOWER]: Refusing update from remote peer 2c8ded37b4784621abda5ddb50b4badd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:00.741143 8528 raft_consensus.cc:1261] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 1 FOLLOWER]: Refusing update from remote peer 2c8ded37b4784621abda5ddb50b4badd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:00.741456 8725 consensus_queue.cc:1035] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [LEADER]: Connected to new peer: Peer: permanent_uuid: "924c79a77c6f442c8af73cfa8ff6db77" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:00.741586 8732 consensus_queue.cc:1035] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c2d2f41d7d414381f0a005279f80d3" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:00.743832 8735 mvcc.cc:205] Tried to move back new op lower bound from 7022712916952997888 to 7022712916878249984. Current Snapshot: MvccSnapshot[applied={T|T < 7022712916952997888}]
I20240501 02:12:00.744226 8737 mvcc.cc:205] Tried to move back new op lower bound from 7022712916952997888 to 7022712916878249984. Current Snapshot: MvccSnapshot[applied={T|T < 7022712916952997888}]
I20240501 02:12:00.744571 8736 mvcc.cc:205] Tried to move back new op lower bound from 7022712916952997888 to 7022712916878249984. Current Snapshot: MvccSnapshot[applied={T|T < 7022712916952997888 or (T in {7022712916952997888})}]
I20240501 02:12:00.746627 8247 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45310:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:00.746733 8247 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45310:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:00.747471 8247 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d2ec58052e474fa0a25c5af260d8a6e2: Sending DeleteTablet for 3 replicas of tablet bc3cee417b8d483487008c57474467ee
I20240501 02:12:00.747792 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3cee417b8d483487008c57474467ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:00 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:56120
I20240501 02:12:00.747797 8374 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3cee417b8d483487008c57474467ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:00 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:53158
I20240501 02:12:00.747839 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3cee417b8d483487008c57474467ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:00 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:40192
I20240501 02:12:00.748023 8745 tablet_replica.cc:330] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd: stopping tablet replica
I20240501 02:12:00.748065 8747 tablet_replica.cc:330] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: stopping tablet replica
I20240501 02:12:00.748138 8745 raft_consensus.cc:2229] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:12:00.748203 8747 raft_consensus.cc:2229] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:00.748364 8747 raft_consensus.cc:2258] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:00.748412 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8311
I20240501 02:12:00.748436 8745 raft_consensus.cc:2258] T bc3cee417b8d483487008c57474467ee P 2c8ded37b4784621abda5ddb50b4badd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:00.749682 8746 tablet_replica.cc:330] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: stopping tablet replica
I20240501 02:12:00.749810 8746 raft_consensus.cc:2229] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:00.749918 8746 raft_consensus.cc:2258] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:00.750599 8747 ts_tablet_manager.cc:1875] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:00.750599 8746 ts_tablet_manager.cc:1875] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:00.752364 8746 ts_tablet_manager.cc:1888] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:00.752364 8747 ts_tablet_manager.cc:1888] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:00.752439 8747 log.cc:1196] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/bc3cee417b8d483487008c57474467ee
I20240501 02:12:00.752439 8746 log.cc:1196] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/bc3cee417b8d483487008c57474467ee
I20240501 02:12:00.752718 8746 ts_tablet_manager.cc:1909] T bc3cee417b8d483487008c57474467ee P 49c2d2f41d7d414381f0a005279f80d3: Deleting consensus metadata
I20240501 02:12:00.752718 8747 ts_tablet_manager.cc:1909] T bc3cee417b8d483487008c57474467ee P 924c79a77c6f442c8af73cfa8ff6db77: Deleting consensus metadata
I20240501 02:12:00.753106 8234 catalog_manager.cc:4830] TS 924c79a77c6f442c8af73cfa8ff6db77 (127.22.12.67:36579): tablet bc3cee417b8d483487008c57474467ee (table test-table [id=e0f9def632054ad0950394fd35ea429f]) successfully deleted
W20240501 02:12:00.753235 8234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:33859 (error 108)
I20240501 02:12:00.753420 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8445
I20240501 02:12:00.753114 8235 catalog_manager.cc:4830] TS 49c2d2f41d7d414381f0a005279f80d3 (127.22.12.66:40619): tablet bc3cee417b8d483487008c57474467ee (table test-table [id=e0f9def632054ad0950394fd35ea429f]) successfully deleted
I20240501 02:12:00.753998 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.753544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:33859 (local address 127.0.0.1:53172)
0501 02:12:00.753930 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:00.753965 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20240501 02:12:00.754667 8234 catalog_manager.cc:4573] TS 2c8ded37b4784621abda5ddb50b4badd (127.22.12.65:33859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc3cee417b8d483487008c57474467ee: Network error: Client connection negotiation failed: client connection to 127.22.12.65:33859: connect: Connection refused (error 111)
I20240501 02:12:00.757508 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8579
I20240501 02:12:00.761802 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8217
2024-05-01T02:12:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1718 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2024-05-01T02:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:00Z Disabled control of system clock
I20240501 02:12:00.787305 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:38609
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:38609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:12:00.865319 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:00.865507 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:00.865526 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:00.866823 8752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:00.866865 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:00.866876 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:00.866887 8752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:00.866896 8752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:00.868317 8752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:38609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:00.868510 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:00.868721 8752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:00.872817 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.873205 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:00.873363 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:00.873780 8752 server_base.cc:1034] running on GCE node
I20240501 02:12:00.874017 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:00.874333 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:00.875473 8752 hybrid_clock.cc:648] HybridClock initialized: now 1714529520875436 us; error 55 us; skew 500 ppm
I20240501 02:12:00.876694 8752 webserver.cc:460] Webserver started at http://127.22.12.126:39063/ using document root <none> and password file <none>
I20240501 02:12:00.876889 8752 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:00.876931 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:00.877038 8752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:00.878068 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "318066ad697747eaa2c90fbe004bb384"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "1faa8aef58dc1d0cb2d830e8c59148d4"
server_key_iv: "7e5287ace2f5286da6fbdbb52afbb6dc"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.878371 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "318066ad697747eaa2c90fbe004bb384"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "1faa8aef58dc1d0cb2d830e8c59148d4"
server_key_iv: "7e5287ace2f5286da6fbdbb52afbb6dc"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.879599 8752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:00.880291 8766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:00.880463 8752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:00.880540 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "318066ad697747eaa2c90fbe004bb384"
format_stamp: "Formatted at 2024-05-01 02:12:00 on dist-test-slave-9tkh"
server_key: "1faa8aef58dc1d0cb2d830e8c59148d4"
server_key_iv: "7e5287ace2f5286da6fbdbb52afbb6dc"
server_key_version: "encryptionkey@0"
I20240501 02:12:00.880601 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:00.892515 8752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:00.892815 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:00.892925 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:00.896518 8752 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:38609
I20240501 02:12:00.896562 8818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:38609 every 8 connection(s)
I20240501 02:12:00.896864 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:00.897476 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:00.900034 8819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: Bootstrap starting.
I20240501 02:12:00.900619 8819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:00.900707 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8752
I20240501 02:12:00.900843 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:00.900925 8819 log.cc:824] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:00.901023 22577 external_mini_cluster.cc:1428] Setting key 3580a0c572f6372698f21ac2efbb62fe
I20240501 02:12:00.901739 8819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: No bootstrap required, opened a new log
I20240501 02:12:00.903502 8819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318066ad697747eaa2c90fbe004bb384" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 38609 } }
I20240501 02:12:00.903632 8819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:00.903649 8819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318066ad697747eaa2c90fbe004bb384, State: Initialized, Role: FOLLOWER
I20240501 02:12:00.903738 8819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318066ad697747eaa2c90fbe004bb384" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 38609 } }
I20240501 02:12:00.903803 8819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:00.903889 8819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:00.903941 8819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:00.904471 8819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318066ad697747eaa2c90fbe004bb384" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 38609 } }
I20240501 02:12:00.904582 8819 leader_election.cc:302] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 318066ad697747eaa2c90fbe004bb384; no voters:
I20240501 02:12:00.904731 8819 leader_election.cc:288] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:00.904925 8824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:00.904965 8819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:00.905120 8824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [term 1 LEADER]: Becoming Leader. State: Replica: 318066ad697747eaa2c90fbe004bb384, State: Running, Role: LEADER
I20240501 02:12:00.905246 8824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318066ad697747eaa2c90fbe004bb384" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 38609 } }
I20240501 02:12:00.905768 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:00.901593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.22.12.126:38609)
0501 02:12:00.901789 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:00.901797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:00.901820 (+ 23us) server_negotiation.cc:407] Connection header received
0501 02:12:00.902283 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:00.902290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:00.902343 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:00.902461 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:00.902959 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.903742 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:00.904359 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:00.904608 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:00.905121 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:00.905130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:00.905130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:00.905309 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:00.905346 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:00.905355 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:00.905425 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:00.905517 (+ 92us) server_negotiation.cc:299] Negotiation successful
0501 02:12:00.905570 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20240501 02:12:00.906319 8826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318066ad697747eaa2c90fbe004bb384. Latest consensus state: current_term: 1 leader_uuid: "318066ad697747eaa2c90fbe004bb384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318066ad697747eaa2c90fbe004bb384" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 38609 } } }
I20240501 02:12:00.906402 8826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:00.906594 8825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "318066ad697747eaa2c90fbe004bb384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318066ad697747eaa2c90fbe004bb384" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 38609 } } }
I20240501 02:12:00.906826 8825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384 [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:00.906978 8839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:00.907013 8839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:00.907073 8840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:00.907264 8840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:00.908541 8840 catalog_manager.cc:1347] Generated new cluster ID: 55a5bdf98533496aa4303064457a0b77
I20240501 02:12:00.908589 8840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:00.929356 8840 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:00.929911 8840 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:00.942009 8840 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: Generated new TSK 0
I20240501 02:12:00.942230 8840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:00.945642 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:38609
--builtin_ntp_servers=127.22.12.84:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:12:01.025133 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:01.025327 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:01.025349 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:01.026746 8843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:01.026790 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:01.026815 8843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:12:01.028249 8843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:38609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:01.028452 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:01.028694 8843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:01.033380 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:01.033710 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:01.034060 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:01.034457 8843 server_base.cc:1034] running on GCE node
I20240501 02:12:01.034672 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:01.034888 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:01.036028 8843 hybrid_clock.cc:648] HybridClock initialized: now 1714529521036016 us; error 33 us; skew 500 ppm
I20240501 02:12:01.042668 8843 webserver.cc:460] Webserver started at http://127.22.12.65:45705/ using document root <none> and password file <none>
I20240501 02:12:01.042902 8843 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:01.042950 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:01.043095 8843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:01.044183 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "6be5e224653044fba0275a84cd7aaf24"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "1e1811abee22fe2d5083bcdd1c7aa29f"
server_key_iv: "08ddb61c0066bc948f5ba43b85882e32"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.044495 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "6be5e224653044fba0275a84cd7aaf24"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "1e1811abee22fe2d5083bcdd1c7aa29f"
server_key_iv: "08ddb61c0066bc948f5ba43b85882e32"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.045683 8843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:12:01.046363 8857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.046594 8843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:01.046701 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "6be5e224653044fba0275a84cd7aaf24"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "1e1811abee22fe2d5083bcdd1c7aa29f"
server_key_iv: "08ddb61c0066bc948f5ba43b85882e32"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.046789 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:01.068414 8843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:01.068696 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:01.068781 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:01.068962 8843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:01.069269 8843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:01.069301 8843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.069334 8843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:01.069355 8843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.074817 8843 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:46199
I20240501 02:12:01.074855 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:46199 every 8 connection(s)
I20240501 02:12:01.075198 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:12:01.079684 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.075908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:54605 (local address 127.22.12.126:38609)
0501 02:12:01.075962 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.075964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.076176 (+ 212us) server_negotiation.cc:407] Connection header received
0501 02:12:01.076709 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.076711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.076721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.076786 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:01.077266 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.077761 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.078258 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.078372 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.079369 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:01.079375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:01.079375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:01.079379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:01.079395 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:01.079399 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:01.079453 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.079591 (+ 138us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.079618 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:12:01.079823 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.075872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:38609 (local address 127.22.12.65:54605)
0501 02:12:01.076061 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.076084 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.076590 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.076848 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.076852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.076860 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:01.077200 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.077205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.077827 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.077828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.078202 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.078203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.078318 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.079242 (+ 924us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:12:01.079242 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:01.079262 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:12:01.079282 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:01.079481 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:01.079486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:01.079489 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.079572 (+ 83us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.079639 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20240501 02:12:01.080366 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8843
I20240501 02:12:01.080377 8971 heartbeater.cc:347] Connected to a master server at 127.22.12.126:38609
I20240501 02:12:01.080476 8971 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:01.080482 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:12:01.080668 22577 external_mini_cluster.cc:1428] Setting key 34323b81c408d4077aa996f7365088b5
I20240501 02:12:01.080735 8971 heartbeater.cc:511] Master 127.22.12.126:38609 requested a full tablet report, sending...
I20240501 02:12:01.081246 8782 ts_manager.cc:194] Registered new tserver with Master: 6be5e224653044fba0275a84cd7aaf24 (127.22.12.65:46199)
I20240501 02:12:01.081570 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:38609
--builtin_ntp_servers=127.22.12.84:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 02:12:01.081982 8782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.65:54605
W20240501 02:12:01.161006 8974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:01.161201 8974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:01.161221 8974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:01.162590 8974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:01.162635 8974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:01.162659 8974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:12:01.164114 8974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:38609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:01.164330 8974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:01.164593 8974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:01.168824 8979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:01.169036 8980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:01.169421 8982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:01.169777 8974 server_base.cc:1034] running on GCE node
I20240501 02:12:01.169975 8974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:01.170190 8974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:01.171330 8974 hybrid_clock.cc:648] HybridClock initialized: now 1714529521171297 us; error 50 us; skew 500 ppm
I20240501 02:12:01.172782 8974 webserver.cc:460] Webserver started at http://127.22.12.66:34187/ using document root <none> and password file <none>
I20240501 02:12:01.172994 8974 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:01.173039 8974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:01.173151 8974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:01.174158 8974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "941bea4e7d9d439d9fc4d39e62886d6e"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "3592e71a53bf99912fcbe0d0ec4a896c"
server_key_iv: "c51890c6a73cee0dae64f27c143164ef"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.174458 8974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "941bea4e7d9d439d9fc4d39e62886d6e"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "3592e71a53bf99912fcbe0d0ec4a896c"
server_key_iv: "c51890c6a73cee0dae64f27c143164ef"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.175666 8974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:01.176399 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.176571 8974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:01.176649 8974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "941bea4e7d9d439d9fc4d39e62886d6e"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "3592e71a53bf99912fcbe0d0ec4a896c"
server_key_iv: "c51890c6a73cee0dae64f27c143164ef"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.176713 8974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:01.190310 8974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:01.190680 8974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:01.190825 8974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:01.191053 8974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:01.191388 8974 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:01.191423 8974 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.191454 8974 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:01.191470 8974 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.197386 8974 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:42005
I20240501 02:12:01.197428 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:42005 every 8 connection(s)
I20240501 02:12:01.197732 8974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:12:01.201910 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.198403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:54033 (local address 127.22.12.126:38609)
0501 02:12:01.198463 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.198466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.198701 (+ 235us) server_negotiation.cc:407] Connection header received
0501 02:12:01.199151 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.199152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.199161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.199216 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:01.199699 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.200188 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.200702 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.200818 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.201679 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:01.201683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:01.201683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:01.201687 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:01.201703 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:01.201706 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:01.201752 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.201819 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.201847 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:12:01.202013 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.198384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:38609 (local address 127.22.12.66:54033)
0501 02:12:01.198602 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.198628 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.199072 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.199274 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.199277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.199285 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:01.199602 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.199605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.200261 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.200262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.200629 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.200630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.200726 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.201559 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:12:01.201559 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:01.201577 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:12:01.201597 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:01.201761 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:01.201764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:01.201766 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.201817 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.201875 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20240501 02:12:01.202479 9102 heartbeater.cc:347] Connected to a master server at 127.22.12.126:38609
I20240501 02:12:01.202636 9102 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:01.202836 9102 heartbeater.cc:511] Master 127.22.12.126:38609 requested a full tablet report, sending...
I20240501 02:12:01.203274 8782 ts_manager.cc:194] Registered new tserver with Master: 941bea4e7d9d439d9fc4d39e62886d6e (127.22.12.66:42005)
I20240501 02:12:01.203691 8782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.66:54033
I20240501 02:12:01.206035 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 8974
I20240501 02:12:01.206133 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:12:01.206305 22577 external_mini_cluster.cc:1428] Setting key 1fb8cd307995b3bb05e1cafac660a346
I20240501 02:12:01.207304 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:38609
--builtin_ntp_servers=127.22.12.84:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:12:01.286319 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:01.286509 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:01.286556 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:01.287884 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:01.287927 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:01.287951 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:12:01.289340 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:38609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:01.289518 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:01.289738 9105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:01.294292 9110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:01.294378 9111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:01.294831 9113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:01.295202 9105 server_base.cc:1034] running on GCE node
I20240501 02:12:01.295385 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:01.295607 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:01.296761 9105 hybrid_clock.cc:648] HybridClock initialized: now 1714529521296725 us; error 49 us; skew 500 ppm
I20240501 02:12:01.298101 9105 webserver.cc:460] Webserver started at http://127.22.12.67:41771/ using document root <none> and password file <none>
I20240501 02:12:01.298290 9105 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:01.298336 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:01.298446 9105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:01.299486 9105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "fb8cd26a953c44dab3696756b8d0bb65"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "3b15c757ebb8c15272757320742f885b"
server_key_iv: "aff3388f92356a940e799ddabaf9d604"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.299778 9105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "fb8cd26a953c44dab3696756b8d0bb65"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "3b15c757ebb8c15272757320742f885b"
server_key_iv: "aff3388f92356a940e799ddabaf9d604"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.300897 9105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:01.301646 9119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.301841 9105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:01.301926 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "fb8cd26a953c44dab3696756b8d0bb65"
format_stamp: "Formatted at 2024-05-01 02:12:01 on dist-test-slave-9tkh"
server_key: "3b15c757ebb8c15272757320742f885b"
server_key_iv: "aff3388f92356a940e799ddabaf9d604"
server_key_version: "encryptionkey@0"
I20240501 02:12:01.301988 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:01.325650 9105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:01.325945 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:01.326052 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:01.326269 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:01.326643 9105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:01.326679 9105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.326709 9105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:01.326731 9105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:01.332088 9105 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:42889
I20240501 02:12:01.332146 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:42889 every 8 connection(s)
I20240501 02:12:01.332486 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:12:01.336911 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.333152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:38919 (local address 127.22.12.126:38609)
0501 02:12:01.333202 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.333204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.333447 (+ 243us) server_negotiation.cc:407] Connection header received
0501 02:12:01.333915 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.333916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.333926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.333983 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:01.334468 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.334982 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.335516 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.335633 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.336609 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:01.336614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:01.336614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:01.336619 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:01.336635 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:01.336639 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:01.336687 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.336816 (+ 129us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.336846 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:12:01.337076 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.333124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:38609 (local address 127.22.12.67:38919)
0501 02:12:01.333339 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.333364 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.333828 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.334046 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.334050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.334058 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:01.334368 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.334371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.335064 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.335065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.335446 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.335447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.335557 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.336471 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:12:01.336472 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:01.336491 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:12:01.336517 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:01.336726 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:01.336732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:01.336734 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.336829 (+ 95us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.336899 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20240501 02:12:01.337631 9233 heartbeater.cc:347] Connected to a master server at 127.22.12.126:38609
I20240501 02:12:01.337743 9233 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:01.337950 9233 heartbeater.cc:511] Master 127.22.12.126:38609 requested a full tablet report, sending...
I20240501 02:12:01.338371 8782 ts_manager.cc:194] Registered new tserver with Master: fb8cd26a953c44dab3696756b8d0bb65 (127.22.12.67:42889)
I20240501 02:12:01.338766 8782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.67:38919
I20240501 02:12:01.341800 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9105
I20240501 02:12:01.341893 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:12:01.342056 22577 external_mini_cluster.cc:1428] Setting key 113fed7dc192eb78585f590a5e05a271
I20240501 02:12:01.342957 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:12:01.346837 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.344175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.22.12.126:38609)
0501 02:12:01.344216 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.344218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.344333 (+ 115us) server_negotiation.cc:407] Connection header received
0501 02:12:01.344395 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.344396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.344405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.344450 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:01.344901 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.345363 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.345898 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.346054 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.346510 (+ 456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:01.346514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:01.346514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:01.346519 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:01.346565 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:01.346569 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:01.346657 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.346747 (+ 90us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.346758 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:12:01.348340 8782 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:12:01.349694 8782 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:12:01.354334 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.351692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:42889 (local address 127.0.0.1:58442)
0501 02:12:01.352037 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.352049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.352123 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.352325 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.352329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.352331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:01.352529 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.352531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.353486 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.353487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.354066 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.354068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.354130 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.354169 (+ 39us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.354218 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.354261 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":68,"threads_started":1}
I20240501 02:12:01.354892 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.351786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.22.12.67:42889)
0501 02:12:01.351958 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.351966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.352193 (+ 227us) server_negotiation.cc:407] Connection header received
0501 02:12:01.352229 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.352231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.352245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.352305 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:01.352615 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.353408 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.354148 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.354663 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.354754 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.354787 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.354825 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20240501 02:12:01.355345 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.351887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:46199 (local address 127.0.0.1:34854)
0501 02:12:01.352168 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.352176 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.352230 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.352468 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.352469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.352472 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:01.352731 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.352732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.354073 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.354074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.355149 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.355151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.355213 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.355216 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.355244 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.355278 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":71,"threads_started":1}
I20240501 02:12:01.355422 9167 tablet_service.cc:1467] Processing CreateTablet for tablet d4c0422addc646ebb1d52de83eb7df78 (DEFAULT_TABLE table=test-table [id=8f0c0311db6c4d1989b3e5d859ed772b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:01.355742 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c0422addc646ebb1d52de83eb7df78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:01.355839 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.351683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:42005 (local address 127.0.0.1:52942)
0501 02:12:01.352021 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.352046 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.352286 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.352546 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.352546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.352548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:01.352730 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.352731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.354075 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.354076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.355000 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.355003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.355683 (+ 680us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.355687 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.355741 (+ 54us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.355780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":72,"threads_started":1}
I20240501 02:12:01.356063 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.351774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.22.12.66:42005)
0501 02:12:01.351958 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.351966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.352117 (+ 151us) server_negotiation.cc:407] Connection header received
0501 02:12:01.352369 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.352371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.352387 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.352433 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:01.352815 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.353972 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.355102 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.355577 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.355656 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.355772 (+ 116us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.356007 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20240501 02:12:01.356088 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.351986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.22.12.65:46199)
0501 02:12:01.352169 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.352177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.352218 (+ 41us) server_negotiation.cc:407] Connection header received
0501 02:12:01.352295 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.352297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.352313 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.352387 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:01.352807 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.353994 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.355225 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.355876 (+ 651us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.355949 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.355980 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.356015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20240501 02:12:01.356565 8905 tablet_service.cc:1467] Processing CreateTablet for tablet d4c0422addc646ebb1d52de83eb7df78 (DEFAULT_TABLE table=test-table [id=8f0c0311db6c4d1989b3e5d859ed772b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:01.356629 9023 tablet_service.cc:1467] Processing CreateTablet for tablet d4c0422addc646ebb1d52de83eb7df78 (DEFAULT_TABLE table=test-table [id=8f0c0311db6c4d1989b3e5d859ed772b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:01.356806 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c0422addc646ebb1d52de83eb7df78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:01.356935 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c0422addc646ebb1d52de83eb7df78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:01.358706 9247 tablet_bootstrap.cc:492] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Bootstrap starting.
I20240501 02:12:01.359215 9248 tablet_bootstrap.cc:492] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24: Bootstrap starting.
I20240501 02:12:01.359508 9247 tablet_bootstrap.cc:654] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:01.359680 9249 tablet_bootstrap.cc:492] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Bootstrap starting.
I20240501 02:12:01.359763 9248 tablet_bootstrap.cc:654] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:01.360044 9248 log.cc:824] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:01.360141 9247 log.cc:824] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:01.360189 9249 tablet_bootstrap.cc:654] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:01.360433 9249 log.cc:824] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:01.360860 9247 tablet_bootstrap.cc:492] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: No bootstrap required, opened a new log
I20240501 02:12:01.360925 9247 ts_tablet_manager.cc:1367] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:01.361029 9249 tablet_bootstrap.cc:492] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: No bootstrap required, opened a new log
I20240501 02:12:01.361039 9248 tablet_bootstrap.cc:492] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24: No bootstrap required, opened a new log
I20240501 02:12:01.361099 9249 ts_tablet_manager.cc:1367] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:01.361099 9248 ts_tablet_manager.cc:1367] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:01.362510 9247 raft_consensus.cc:348] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.362666 9247 raft_consensus.cc:374] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:01.362689 9247 raft_consensus.cc:729] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb8cd26a953c44dab3696756b8d0bb65, State: Initialized, Role: FOLLOWER
I20240501 02:12:01.362803 9247 consensus_queue.cc:260] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.363022 9247 ts_tablet_manager.cc:1398] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 02:12:01.363054 9233 heartbeater.cc:503] Master 127.22.12.126:38609 was elected leader, sending a full tablet report...
I20240501 02:12:01.363289 9249 raft_consensus.cc:348] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.363287 9248 raft_consensus.cc:348] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.363415 9249 raft_consensus.cc:374] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:01.363415 9248 raft_consensus.cc:374] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:01.363440 9248 raft_consensus.cc:729] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be5e224653044fba0275a84cd7aaf24, State: Initialized, Role: FOLLOWER
I20240501 02:12:01.363440 9249 raft_consensus.cc:729] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941bea4e7d9d439d9fc4d39e62886d6e, State: Initialized, Role: FOLLOWER
I20240501 02:12:01.363519 9248 consensus_queue.cc:260] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.363519 9249 consensus_queue.cc:260] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.363731 9248 ts_tablet_manager.cc:1398] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 02:12:01.363732 9249 ts_tablet_manager.cc:1398] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 02:12:01.363840 9102 heartbeater.cc:503] Master 127.22.12.126:38609 was elected leader, sending a full tablet report...
I20240501 02:12:01.364118 8971 heartbeater.cc:503] Master 127.22.12.126:38609 was elected leader, sending a full tablet report...
I20240501 02:12:01.390362 9254 raft_consensus.cc:482] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:01.390489 9254 raft_consensus.cc:504] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.390826 9254 leader_election.cc:288] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 941bea4e7d9d439d9fc4d39e62886d6e (127.22.12.66:42005), fb8cd26a953c44dab3696756b8d0bb65 (127.22.12.67:42889)
I20240501 02:12:01.392992 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.391024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:42005 (local address 127.22.12.65:36127)
0501 02:12:01.391101 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.391114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.391191 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.391316 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.391317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.391318 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:01.391486 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.391487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.392135 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.392136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.392759 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.392762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.392836 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.392841 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.392869 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.392912 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20240501 02:12:01.393669 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.391097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:36127 (local address 127.22.12.66:42005)
0501 02:12:01.391145 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.391147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.391158 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:12:01.391252 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.391253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.391262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.391308 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:01.391548 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.392047 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.392868 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.393532 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.393540 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.393562 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.393598 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240501 02:12:01.393694 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.391024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:42889 (local address 127.22.12.65:50939)
0501 02:12:01.391209 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.391218 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:01.391287 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:01.391404 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:01.391405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:01.391406 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:01.391605 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.391607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.392220 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.392221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:01.392894 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:01.392896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.393489 (+ 593us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.393493 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:12:01.393576 (+ 83us) client_negotiation.cc:240] Negotiation successful
0501 02:12:01.393612 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":13,"thread_start_us":99,"threads_started":1}
I20240501 02:12:01.393728 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.391105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:50939 (local address 127.22.12.67:42889)
0501 02:12:01.391154 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.391173 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.391285 (+ 112us) server_negotiation.cc:407] Connection header received
0501 02:12:01.391342 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.391343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.391353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.391395 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:01.391674 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.392145 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.392997 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.393437 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.393445 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.393621 (+ 176us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.393667 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:12:01.394021 9187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c0422addc646ebb1d52de83eb7df78" candidate_uuid: "6be5e224653044fba0275a84cd7aaf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb8cd26a953c44dab3696756b8d0bb65" is_pre_election: true
I20240501 02:12:01.394022 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c0422addc646ebb1d52de83eb7df78" candidate_uuid: "6be5e224653044fba0275a84cd7aaf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" is_pre_election: true
I20240501 02:12:01.394160 9056 raft_consensus.cc:2454] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be5e224653044fba0275a84cd7aaf24 in term 0.
I20240501 02:12:01.394168 9187 raft_consensus.cc:2454] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be5e224653044fba0275a84cd7aaf24 in term 0.
I20240501 02:12:01.394378 8860 leader_election.cc:302] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be5e224653044fba0275a84cd7aaf24, fb8cd26a953c44dab3696756b8d0bb65; no voters:
I20240501 02:12:01.394553 9254 raft_consensus.cc:2781] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:01.394629 9254 raft_consensus.cc:482] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:01.394647 9254 raft_consensus.cc:3037] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:01.395284 9254 raft_consensus.cc:504] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.395403 9254 leader_election.cc:288] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [CANDIDATE]: Term 1 election: Requested vote from peers 941bea4e7d9d439d9fc4d39e62886d6e (127.22.12.66:42005), fb8cd26a953c44dab3696756b8d0bb65 (127.22.12.67:42889)
I20240501 02:12:01.395574 9187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c0422addc646ebb1d52de83eb7df78" candidate_uuid: "6be5e224653044fba0275a84cd7aaf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb8cd26a953c44dab3696756b8d0bb65"
I20240501 02:12:01.395576 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c0422addc646ebb1d52de83eb7df78" candidate_uuid: "6be5e224653044fba0275a84cd7aaf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941bea4e7d9d439d9fc4d39e62886d6e"
I20240501 02:12:01.395642 9187 raft_consensus.cc:3037] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:01.395668 9056 raft_consensus.cc:3037] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:01.396275 9187 raft_consensus.cc:2454] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be5e224653044fba0275a84cd7aaf24 in term 1.
I20240501 02:12:01.396381 9056 raft_consensus.cc:2454] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be5e224653044fba0275a84cd7aaf24 in term 1.
I20240501 02:12:01.396441 8860 leader_election.cc:302] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6be5e224653044fba0275a84cd7aaf24, fb8cd26a953c44dab3696756b8d0bb65; no voters:
I20240501 02:12:01.396543 9254 raft_consensus.cc:2781] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:01.396703 9254 raft_consensus.cc:686] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [term 1 LEADER]: Becoming Leader. State: Replica: 6be5e224653044fba0275a84cd7aaf24, State: Running, Role: LEADER
I20240501 02:12:01.396782 9254 consensus_queue.cc:237] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } }
I20240501 02:12:01.397581 8782 catalog_manager.cc:5484] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6be5e224653044fba0275a84cd7aaf24 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "6be5e224653044fba0275a84cd7aaf24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be5e224653044fba0275a84cd7aaf24" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 46199 } health_report { overall_health: HEALTHY } } }
I20240501 02:12:01.413645 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.411654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.22.12.65:46199)
0501 02:12:01.411717 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:01.411719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:01.411728 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:12:01.411771 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:01.411771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:01.411789 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:01.411860 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:01.412078 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.412571 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:01.413130 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:01.413243 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:01.413384 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:01.413462 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:01.413532 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:01.413557 (+ 25us) server_negotiation.cc:299] Negotiation successful
0501 02:12:01.413565 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 02:12:01.415443 9056 raft_consensus.cc:1261] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 1 FOLLOWER]: Refusing update from remote peer 6be5e224653044fba0275a84cd7aaf24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:01.415443 9187 raft_consensus.cc:1261] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 1 FOLLOWER]: Refusing update from remote peer 6be5e224653044fba0275a84cd7aaf24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:01.415830 9257 consensus_queue.cc:1035] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "941bea4e7d9d439d9fc4d39e62886d6e" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 42005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:01.415979 9254 consensus_queue.cc:1035] T d4c0422addc646ebb1d52de83eb7df78 P 6be5e224653044fba0275a84cd7aaf24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb8cd26a953c44dab3696756b8d0bb65" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:01.418413 9262 mvcc.cc:205] Tried to move back new op lower bound from 7022712919715024896 to 7022712919641747456. Current Snapshot: MvccSnapshot[applied={T|T < 7022712919715024896}]
I20240501 02:12:01.418841 9261 mvcc.cc:205] Tried to move back new op lower bound from 7022712919715024896 to 7022712919641747456. Current Snapshot: MvccSnapshot[applied={T|T < 7022712919715024896 or (T in {7022712919715024896})}]
I20240501 02:12:01.420790 8781 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54040:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:01.420877 8781 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54040:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:01.421609 8781 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 318066ad697747eaa2c90fbe004bb384: Sending DeleteTablet for 3 replicas of tablet d4c0422addc646ebb1d52de83eb7df78
I20240501 02:12:01.421943 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0422addc646ebb1d52de83eb7df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:01 UTC) from {username='slave'} at 127.0.0.1:34854
I20240501 02:12:01.421931 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0422addc646ebb1d52de83eb7df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:01 UTC) from {username='slave'} at 127.0.0.1:58442
I20240501 02:12:01.422156 9023 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0422addc646ebb1d52de83eb7df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:01 UTC) from {username='slave'} at 127.0.0.1:52942
I20240501 02:12:01.422354 9272 tablet_replica.cc:330] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: stopping tablet replica
I20240501 02:12:01.422401 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8843
I20240501 02:12:01.422434 9272 raft_consensus.cc:2229] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:01.423454 9272 raft_consensus.cc:2258] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:01.423892 9272 ts_tablet_manager.cc:1875] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:01.425477 9272 ts_tablet_manager.cc:1888] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:01.425546 9272 log.cc:1196] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/d4c0422addc646ebb1d52de83eb7df78
I20240501 02:12:01.425807 9272 ts_tablet_manager.cc:1909] T d4c0422addc646ebb1d52de83eb7df78 P 941bea4e7d9d439d9fc4d39e62886d6e: Deleting consensus metadata
I20240501 02:12:01.426199 8770 catalog_manager.cc:4830] TS 941bea4e7d9d439d9fc4d39e62886d6e (127.22.12.66:42005): tablet d4c0422addc646ebb1d52de83eb7df78 (table test-table [id=8f0c0311db6c4d1989b3e5d859ed772b]) successfully deleted
W20240501 02:12:01.428361 8769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:46199 (error 108)
I20240501 02:12:01.428551 9271 tablet_replica.cc:330] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: stopping tablet replica
I20240501 02:12:01.428656 9271 raft_consensus.cc:2229] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:01.428772 9271 raft_consensus.cc:2258] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:01.428817 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:01.428683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:46199 (local address 127.0.0.1:34864)
0501 02:12:01.428723 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:01.428766 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20240501 02:12:01.428983 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8974
W20240501 02:12:01.428978 8769 catalog_manager.cc:4573] TS 6be5e224653044fba0275a84cd7aaf24 (127.22.12.65:46199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4c0422addc646ebb1d52de83eb7df78: Network error: Client connection negotiation failed: client connection to 127.22.12.65:46199: connect: Connection refused (error 111)
I20240501 02:12:01.429165 9271 ts_tablet_manager.cc:1875] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:01.430655 9271 ts_tablet_manager.cc:1888] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:01.430730 9271 log.cc:1196] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/d4c0422addc646ebb1d52de83eb7df78
I20240501 02:12:01.431008 9271 ts_tablet_manager.cc:1909] T d4c0422addc646ebb1d52de83eb7df78 P fb8cd26a953c44dab3696756b8d0bb65: Deleting consensus metadata
I20240501 02:12:01.431392 8769 catalog_manager.cc:4830] TS fb8cd26a953c44dab3696756b8d0bb65 (127.22.12.67:42889): tablet d4c0422addc646ebb1d52de83eb7df78 (table test-table [id=8f0c0311db6c4d1989b3e5d859ed772b]) successfully deleted
I20240501 02:12:01.433979 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9105
I20240501 02:12:01.438627 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 8752
2024-05-01T02:12:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (676 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:01 dist-test-slave-9tkh krb5kdc[9276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:01 dist-test-slave-9tkh krb5kdc[9276](info): set up 2 sockets
May 01 02:12:01 dist-test-slave-9tkh krb5kdc[9276](info): commencing operation
krb5kdc: starting...
W20240501 02:12:02.338208 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:02.449849 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:40225
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:40225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:02.528684 9289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:02.528904 9289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:02.528935 9289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:02.530299 9289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:02.530354 9289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:02.530371 9289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:02.530387 9289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:02.530400 9289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:02.532114 9289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:40225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:02.532349 9289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:02.532526 9289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:02.536697 9294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:02.536808 9295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:02.537190 9297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:02.537612 9289 server_base.cc:1034] running on GCE node
I20240501 02:12:02.537830 9289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:02.538058 9289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:02.539187 9289 hybrid_clock.cc:648] HybridClock initialized: now 1714529522539170 us; error 34 us; skew 500 ppm
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:02.541543 9289 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:02.541980 9289 webserver.cc:460] Webserver started at http://127.22.12.126:36029/ using document root <none> and password file <none>
I20240501 02:12:02.542160 9289 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:02.542200 9289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:02.542312 9289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:02.543360 9289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "1ee1d0ffbc3b4a21b628a37039b7a169"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "ed53242ad718fb3d6e2acede9c5e81d9"
server_key_iv: "e127bc40cc67348a79966cf5d0aed9ab"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.543663 9289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "1ee1d0ffbc3b4a21b628a37039b7a169"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "ed53242ad718fb3d6e2acede9c5e81d9"
server_key_iv: "e127bc40cc67348a79966cf5d0aed9ab"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.544814 9289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:12:02.545428 9304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.545579 9289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:02.545656 9289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "1ee1d0ffbc3b4a21b628a37039b7a169"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "ed53242ad718fb3d6e2acede9c5e81d9"
server_key_iv: "e127bc40cc67348a79966cf5d0aed9ab"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.545717 9289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:02.556483 9289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:02.558655 9289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:02.558773 9289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:02.562163 9289 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:40225
I20240501 02:12:02.562206 9356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:40225 every 8 connection(s)
I20240501 02:12:02.562515 9289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:02.563285 9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:02.564201 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9289
I20240501 02:12:02.564291 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:02.564436 22577 external_mini_cluster.cc:1428] Setting key c7790e00fd32d1174400e4f4b674abf3
I20240501 02:12:02.565654 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: Bootstrap starting.
I20240501 02:12:02.566277 9357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:02.566582 9357 log.cc:824] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:02.567241 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: No bootstrap required, opened a new log
I20240501 02:12:02.569203 9357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40225 } }
I20240501 02:12:02.569361 9357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:02.569386 9357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee1d0ffbc3b4a21b628a37039b7a169, State: Initialized, Role: FOLLOWER
I20240501 02:12:02.569504 9357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40225 } }
I20240501 02:12:02.569568 9357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:02.569602 9357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:02.569648 9357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 0 FOLLOWER]: Advancing to term 1
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:02.570171 9357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40225 } }
I20240501 02:12:02.570250 9357 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee1d0ffbc3b4a21b628a37039b7a169; no voters:
I20240501 02:12:02.570402 9357 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:02.570453 9362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:02.570653 9357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:02.570653 9362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee1d0ffbc3b4a21b628a37039b7a169, State: Running, Role: LEADER
I20240501 02:12:02.570886 9362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40225 } }
I20240501 02:12:02.571254 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee1d0ffbc3b4a21b628a37039b7a169. Latest consensus state: current_term: 1 leader_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40225 } } }
I20240501 02:12:02.571278 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee1d0ffbc3b4a21b628a37039b7a169" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 40225 } } }
I20240501 02:12:02.572213 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:02.572247 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169 [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:02.572649 9376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:02.572721 9376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:02.572804 9379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:02.572968 9379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:02.574402 9379 catalog_manager.cc:1347] Generated new cluster ID: 113e262c125b4719bcf46afd833d78f4
I20240501 02:12:02.574474 9379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:02.574515 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:02.564930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.22.12.126:40225)
0501 02:12:02.565075 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:02.565083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:02.565101 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:12:02.565532 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:02.565537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:02.565579 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:02.565697 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:02.566673 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.567551 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:02.568300 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.568470 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:02.570822 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:02.570832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:02.570832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:02.570853 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:02.573102 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:02.573460 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:02.573462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:02.573462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:02.573520 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:02.573685 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:02.573685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:02.573686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:02.573910 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:02.574063 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:02.574162 (+ 99us) server_negotiation.cc:299] Negotiation successful
0501 02:12:02.574293 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20240501 02:12:02.592723 9379 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:02.593289 9379 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:02.599282 9379 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: Generated new TSK 0
I20240501 02:12:02.599494 9379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
Entry for principal kudu/127.22.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab.
I20240501 02:12:02.631408 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40225
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:02.735472 9383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:02.735653 9383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:02.735675 9383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:02.737000 9383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:02.737046 9383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:02.737072 9383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:12:02.738651 9383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.65.keytab
--principal=kudu/127.22.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:02.738864 9383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:02.739046 9383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:02.743182 9388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:02.743417 9389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:02.743762 9391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:02.744202 9383 server_base.cc:1034] running on GCE node
I20240501 02:12:02.744374 9383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:02.744604 9383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:02.745743 9383 hybrid_clock.cc:648] HybridClock initialized: now 1714529522745731 us; error 32 us; skew 500 ppm
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:02.748137 9383 init.cc:377] Logged in from keytab as kudu/127.22.12.65@KRBTEST.COM (short username kudu)
I20240501 02:12:02.748574 9383 webserver.cc:460] Webserver started at http://127.22.12.65:39963/ using document root <none> and password file <none>
I20240501 02:12:02.748755 9383 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:02.748811 9383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:02.748914 9383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:02.749941 9383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "b94f968337694cdba51ec4b44de33808"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "4789aa561dfb8d5f0127449418779806"
server_key_iv: "36fa717d5451b19fca8fcd2fc6b68b97"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.750228 9383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "b94f968337694cdba51ec4b44de33808"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "4789aa561dfb8d5f0127449418779806"
server_key_iv: "36fa717d5451b19fca8fcd2fc6b68b97"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.751400 9383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:12:02.752019 9398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.752180 9383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:02.752252 9383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "b94f968337694cdba51ec4b44de33808"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "4789aa561dfb8d5f0127449418779806"
server_key_iv: "36fa717d5451b19fca8fcd2fc6b68b97"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.752314 9383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:02.779603 9383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:02.781555 9383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:02.781672 9383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:02.781891 9383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:02.782198 9383 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:02.782231 9383 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.782269 9383 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:02.782295 9383 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.787554 9383 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:45151
I20240501 02:12:02.787621 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:45151 every 8 connection(s)
I20240501 02:12:02.787879 9383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:12:02.790232 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9383
I20240501 02:12:02.790324 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:12:02.790472 22577 external_mini_cluster.cc:1428] Setting key 6da3807c37d1a7752b0d6ebe325db22c
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.65@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:02.796499 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:02.788559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:54981 (local address 127.22.12.126:40225)
0501 02:12:02.788616 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:02.788618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:02.788770 (+ 152us) server_negotiation.cc:407] Connection header received
0501 02:12:02.789271 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:02.789272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:02.789284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:02.789343 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:02.790377 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.790994 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:02.791546 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.791714 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:02.793841 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:02.793845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:02.793846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:02.793852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:02.795255 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:02.795706 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:02.795706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:02.795707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:02.795722 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:02.795916 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:02.795916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:02.795916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:02.795999 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:02.796042 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:02.796362 (+ 320us) server_negotiation.cc:299] Negotiation successful
0501 02:12:02.796432 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 02:12:02.796705 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:02.788514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40225 (local address 127.22.12.65:54981)
0501 02:12:02.788671 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:02.788695 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:02.789171 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:02.789412 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:02.789415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:02.789718 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:02.790239 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:02.790244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.791084 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:02.791085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:02.791464 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:02.791465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.791599 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:02.792138 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:02.792155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:02.793739 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:02.795339 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:02.795346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:02.795350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:02.795635 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:02.795776 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:02.795777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:02.795778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:02.795864 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:02.796052 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:02.796056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:02.796227 (+ 171us) client_negotiation.cc:769] Sending connection context
0501 02:12:02.796356 (+ 129us) client_negotiation.cc:240] Negotiation successful
0501 02:12:02.796504 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20240501 02:12:02.797278 9512 heartbeater.cc:347] Connected to a master server at 127.22.12.126:40225
I20240501 02:12:02.797385 9512 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:02.797614 9512 heartbeater.cc:511] Master 127.22.12.126:40225 requested a full tablet report, sending...
I20240501 02:12:02.798188 9321 ts_manager.cc:194] Registered new tserver with Master: b94f968337694cdba51ec4b44de33808 (127.22.12.65:45151)
WARNING: no policy specified for kudu/127.22.12.66@KRBTEST.COM; defaulting to no policy
I20240501 02:12:02.798898 9321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.65@KRBTEST.COM'} at 127.22.12.65:54981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
Entry for principal kudu/127.22.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab.
I20240501 02:12:02.813462 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40225
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:02.891687 9517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:02.891893 9517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:02.891916 9517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:02.893209 9517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:02.893246 9517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:02.893270 9517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:12:02.894806 9517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.66.keytab
--principal=kudu/127.22.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:02.894987 9517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:02.895172 9517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:02.899343 9522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:02.899451 9523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:02.899839 9525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:02.900208 9517 server_base.cc:1034] running on GCE node
I20240501 02:12:02.900388 9517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:02.900592 9517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:02.901789 9517 hybrid_clock.cc:648] HybridClock initialized: now 1714529522901776 us; error 27 us; skew 500 ppm
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:02.904170 9517 init.cc:377] Logged in from keytab as kudu/127.22.12.66@KRBTEST.COM (short username kudu)
I20240501 02:12:02.904561 9517 webserver.cc:460] Webserver started at http://127.22.12.66:32861/ using document root <none> and password file <none>
I20240501 02:12:02.904767 9517 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:02.904807 9517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:02.904907 9517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:02.905876 9517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "31f401d8f9984fcc9509b9905b5c4cf6"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "98e4d4e15b609cbbac7f9b3d9b9d92c1"
server_key_iv: "6f40ccaeec0d2334c7e1737055fd021f"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.906154 9517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "31f401d8f9984fcc9509b9905b5c4cf6"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "98e4d4e15b609cbbac7f9b3d9b9d92c1"
server_key_iv: "6f40ccaeec0d2334c7e1737055fd021f"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.907297 9517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:12:02.907953 9532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.908095 9517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:02.908166 9517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "31f401d8f9984fcc9509b9905b5c4cf6"
format_stamp: "Formatted at 2024-05-01 02:12:02 on dist-test-slave-9tkh"
server_key: "98e4d4e15b609cbbac7f9b3d9b9d92c1"
server_key_iv: "6f40ccaeec0d2334c7e1737055fd021f"
server_key_version: "encryptionkey@0"
I20240501 02:12:02.908231 9517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:02.918128 9517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:02.920106 9517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:02.920212 9517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:02.920401 9517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:02.920681 9517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:02.920720 9517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.920748 9517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:02.920778 9517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:02.925721 9517 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:36303
I20240501 02:12:02.925776 9645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:36303 every 8 connection(s)
I20240501 02:12:02.926043 9517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:12:02.927954 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9517
I20240501 02:12:02.928074 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:12:02.928279 22577 external_mini_cluster.cc:1428] Setting key b2cefecb714ab6918655b117b1b7b8eb
May 01 02:12:02 dist-test-slave-9tkh krb5kdc[9276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529522, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.66@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:02.934370 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:02.926691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:58571 (local address 127.22.12.126:40225)
0501 02:12:02.926763 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:02.926765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:02.926930 (+ 165us) server_negotiation.cc:407] Connection header received
0501 02:12:02.927357 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:02.927358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:02.927368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:02.927418 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:02.928588 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.929138 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:02.929717 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.929881 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:02.931638 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:02.931643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:02.931643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:02.931650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:02.933114 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:02.933552 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:02.933553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:02.933553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:02.933573 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:02.933775 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:02.933775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:02.933775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:02.933861 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:02.933902 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:02.934232 (+ 330us) server_negotiation.cc:299] Negotiation successful
0501 02:12:02.934301 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:12:02.934572 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:02.926677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40225 (local address 127.22.12.66:58571)
0501 02:12:02.926827 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:02.926852 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:02.927276 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:02.927462 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:02.927465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:02.927802 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:02.928490 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:02.928497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.929251 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:02.929252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:02.929641 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:02.929642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:02.929746 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:02.930238 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:02.930253 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:02.931550 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:02.933198 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:02.933205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:02.933208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:02.933483 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:02.933626 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:02.933627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:02.933627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:02.933707 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:02.933911 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:02.933917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:02.934073 (+ 156us) client_negotiation.cc:769] Sending connection context
0501 02:12:02.934202 (+ 129us) client_negotiation.cc:240] Negotiation successful
0501 02:12:02.934348 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20240501 02:12:02.935037 9646 heartbeater.cc:347] Connected to a master server at 127.22.12.126:40225
I20240501 02:12:02.935137 9646 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:02.935334 9646 heartbeater.cc:511] Master 127.22.12.126:40225 requested a full tablet report, sending...
I20240501 02:12:02.935755 9321 ts_manager.cc:194] Registered new tserver with Master: 31f401d8f9984fcc9509b9905b5c4cf6 (127.22.12.66:36303)
I20240501 02:12:02.936131 9321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.66@KRBTEST.COM'} at 127.22.12.66:58571
WARNING: no policy specified for kudu/127.22.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
Entry for principal kudu/127.22.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab.
I20240501 02:12:02.952594 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40225
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:03.031309 9651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:03.031564 9651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:03.031595 9651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:03.032959 9651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:03.033010 9651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:03.033043 9651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:12:03.034686 9651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.67.keytab
--principal=kudu/127.22.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:03.034920 9651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:03.035106 9651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:03.039036 9656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.039410 9657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.039783 9659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:03.040194 9651 server_base.cc:1034] running on GCE node
I20240501 02:12:03.040364 9651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:03.040565 9651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:03.041702 9651 hybrid_clock.cc:648] HybridClock initialized: now 1714529523041691 us; error 23 us; skew 500 ppm
May 01 02:12:03 dist-test-slave-9tkh krb5kdc[9276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529523, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:03.044183 9651 init.cc:377] Logged in from keytab as kudu/127.22.12.67@KRBTEST.COM (short username kudu)
I20240501 02:12:03.044616 9651 webserver.cc:460] Webserver started at http://127.22.12.67:40003/ using document root <none> and password file <none>
I20240501 02:12:03.044790 9651 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:03.044832 9651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:03.044934 9651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:03.045893 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "3f5e4a74cc024e0dbd188df265940d40"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "9494d7dfef91c87ab18b017418aa8666"
server_key_iv: "21aab7a06741a624f119917587061427"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.046185 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "3f5e4a74cc024e0dbd188df265940d40"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "9494d7dfef91c87ab18b017418aa8666"
server_key_iv: "21aab7a06741a624f119917587061427"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.047385 9651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:12:03.048027 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.048183 9651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:03.048292 9651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "3f5e4a74cc024e0dbd188df265940d40"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "9494d7dfef91c87ab18b017418aa8666"
server_key_iv: "21aab7a06741a624f119917587061427"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.048352 9651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:03.062070 9651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:03.064185 9651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:03.064306 9651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:03.064497 9651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:03.064790 9651 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:03.064821 9651 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.064848 9651 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:03.064865 9651 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.070151 9651 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:44141
I20240501 02:12:03.070199 9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:44141 every 8 connection(s)
I20240501 02:12:03.070567 9651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
May 01 02:12:03 dist-test-slave-9tkh krb5kdc[9276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529523, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.67@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:03.076794 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9651
I20240501 02:12:03.076910 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:12:03.077080 22577 external_mini_cluster.cc:1428] Setting key bebefdf5c5bbe2509ba12b5e3280ac4c
I20240501 02:12:03.078752 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.071268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:42501 (local address 127.22.12.126:40225)
0501 02:12:03.071353 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.071355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.071504 (+ 149us) server_negotiation.cc:407] Connection header received
0501 02:12:03.071948 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.071949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.071958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.072029 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.073087 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.073572 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.074077 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.074180 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.076132 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.076136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.076137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:03.076143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.077609 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:03.078073 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:03.078074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:03.078075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:03.078096 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:03.078238 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:03.078238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:03.078239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:03.078329 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.078372 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.078612 (+ 240us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.078687 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 02:12:03.078869 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.071226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:40225 (local address 127.22.12.67:42501)
0501 02:12:03.071396 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.071426 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.071869 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.072092 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.072096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.072429 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:03.072992 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.072997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.073646 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.073646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.074023 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.074025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.074124 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.074671 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:03.074686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:03.076033 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:03.077702 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:03.077709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:03.077712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:03.078009 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:03.078144 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:03.078145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:03.078145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:03.078200 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:03.078368 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:03.078371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:03.078479 (+ 108us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.078590 (+ 111us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.078703 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20240501 02:12:03.079301 9780 heartbeater.cc:347] Connected to a master server at 127.22.12.126:40225
I20240501 02:12:03.079394 9780 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:03.079560 9780 heartbeater.cc:511] Master 127.22.12.126:40225 requested a full tablet report, sending...
I20240501 02:12:03.079885 9321 ts_manager.cc:194] Registered new tserver with Master: 3f5e4a74cc024e0dbd188df265940d40 (127.22.12.67:44141)
I20240501 02:12:03.080241 9321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.12.67@KRBTEST.COM'} at 127.22.12.67:42501
I20240501 02:12:03.088869 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:12:03.095402 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.090158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.22.12.126:40225)
0501 02:12:03.090222 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.090224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.090295 (+ 71us) server_negotiation.cc:407] Connection header received
0501 02:12:03.090360 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.090361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.090371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.090431 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.091157 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.091641 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.092184 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.092346 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.093244 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.093248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.093249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:03.093255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.094406 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:03.094739 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:03.094741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:03.094741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:03.094775 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:03.094907 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:03.094907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:03.094908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:03.095025 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.095115 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.095218 (+ 103us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.095304 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 02:12:03.096905 9321 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:12:03.098336 9321 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:12:03.103344 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.100196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45151 (local address 127.0.0.1:44884)
0501 02:12:03.100527 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.100540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.100602 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.101057 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.101058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.101059 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.101293 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.101296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.102378 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.102379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.103038 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.103039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.103106 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.103151 (+ 45us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.103204 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.103247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":4,"thread_start_us":51,"threads_started":1}
I20240501 02:12:03.103576 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.100195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:44141 (local address 127.0.0.1:34154)
0501 02:12:03.100361 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.100385 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.100445 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.100770 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.100773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.100776 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.101033 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.101035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.102386 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.102387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.103371 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.103374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.103437 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.103442 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.103469 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.103508 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":55,"threads_started":1}
I20240501 02:12:03.104055 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.100350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.22.12.65:45151)
0501 02:12:03.100899 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.100906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.100915 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:12:03.100951 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.100952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.100968 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.101040 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.101395 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.102204 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.103135 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.103899 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.103939 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.103966 (+ 27us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.103997 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":68,"threads_started":1}
I20240501 02:12:03.104423 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.100579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.22.12.66:36303)
0501 02:12:03.100749 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.100763 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.100773 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:12:03.100804 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.100805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.100817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.100888 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.101165 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.102310 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.103719 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.104227 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.104273 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.104305 (+ 32us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.104352 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20240501 02:12:03.104420 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.100294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.22.12.67:44141)
0501 02:12:03.100620 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.100628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.100637 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:12:03.100671 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.100672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.100686 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.100752 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.101124 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.102288 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.103687 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.104226 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.104273 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.104305 (+ 32us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.104345 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
I20240501 02:12:03.104563 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.100402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:36303 (local address 127.0.0.1:55872)
0501 02:12:03.100534 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.100541 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.100587 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.100901 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.100902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.100904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.101098 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.101100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.102394 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.102395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.103372 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.103374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.103555 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.103558 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.103584 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.103615 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":46,"threads_started":1}
I20240501 02:12:03.104547 9446 tablet_service.cc:1467] Processing CreateTablet for tablet 7be92bad8fa540fdb640db9259e76438 (DEFAULT_TABLE table=test-table [id=b07e212e1d6d4034afd146ae3e42a872]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:03.104858 9446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be92bad8fa540fdb640db9259e76438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.104952 9714 tablet_service.cc:1467] Processing CreateTablet for tablet 7be92bad8fa540fdb640db9259e76438 (DEFAULT_TABLE table=test-table [id=b07e212e1d6d4034afd146ae3e42a872]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:03.105211 9714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be92bad8fa540fdb640db9259e76438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.107095 9580 tablet_service.cc:1467] Processing CreateTablet for tablet 7be92bad8fa540fdb640db9259e76438 (DEFAULT_TABLE table=test-table [id=b07e212e1d6d4034afd146ae3e42a872]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:03.107383 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be92bad8fa540fdb640db9259e76438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.107594 9794 tablet_bootstrap.cc:492] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: Bootstrap starting.
I20240501 02:12:03.107892 9795 tablet_bootstrap.cc:492] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Bootstrap starting.
I20240501 02:12:03.108228 9794 tablet_bootstrap.cc:654] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.108393 9795 tablet_bootstrap.cc:654] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.108527 9794 log.cc:824] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.108666 9795 log.cc:824] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.109330 9794 tablet_bootstrap.cc:492] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: No bootstrap required, opened a new log
I20240501 02:12:03.109385 9794 ts_tablet_manager.cc:1367] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:03.109351 9795 tablet_bootstrap.cc:492] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: No bootstrap required, opened a new log
I20240501 02:12:03.109562 9795 ts_tablet_manager.cc:1367] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:03.110051 9798 tablet_bootstrap.cc:492] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Bootstrap starting.
I20240501 02:12:03.110656 9798 tablet_bootstrap.cc:654] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.110967 9798 log.cc:824] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.111579 9794 raft_consensus.cc:348] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.111615 9795 raft_consensus.cc:348] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.111714 9795 raft_consensus.cc:374] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.111716 9794 raft_consensus.cc:374] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.111714 9798 tablet_bootstrap.cc:492] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: No bootstrap required, opened a new log
I20240501 02:12:03.111734 9795 raft_consensus.cc:729] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5e4a74cc024e0dbd188df265940d40, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.111742 9794 raft_consensus.cc:729] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94f968337694cdba51ec4b44de33808, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.111765 9798 ts_tablet_manager.cc:1367] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 02:12:03.111831 9794 consensus_queue.cc:260] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.111833 9795 consensus_queue.cc:260] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.112061 9794 ts_tablet_manager.cc:1398] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:12:03.112064 9795 ts_tablet_manager.cc:1398] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 02:12:03.112164 9780 heartbeater.cc:503] Master 127.22.12.126:40225 was elected leader, sending a full tablet report...
I20240501 02:12:03.112387 9512 heartbeater.cc:503] Master 127.22.12.126:40225 was elected leader, sending a full tablet report...
I20240501 02:12:03.113796 9798 raft_consensus.cc:348] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.113909 9798 raft_consensus.cc:374] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.113924 9798 raft_consensus.cc:729] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f401d8f9984fcc9509b9905b5c4cf6, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.114010 9798 consensus_queue.cc:260] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.114209 9798 ts_tablet_manager.cc:1398] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 02:12:03.114267 9646 heartbeater.cc:503] Master 127.22.12.126:40225 was elected leader, sending a full tablet report...
W20240501 02:12:03.176525 9647 tablet.cc:2369] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 02:12:03.288529 9513 tablet.cc:2369] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:12:03.302801 9800 raft_consensus.cc:482] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:03.302927 9800 raft_consensus.cc:504] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.303218 9800 leader_election.cc:288] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f5e4a74cc024e0dbd188df265940d40 (127.22.12.67:44141), 31f401d8f9984fcc9509b9905b5c4cf6 (127.22.12.66:36303)
I20240501 02:12:03.306313 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.303422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:36303 (local address 127.22.12.65:34911)
0501 02:12:03.303545 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.303557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.303634 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.303777 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.303777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.303793 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.304119 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.304122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.305100 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.305101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.306038 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.306041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.306122 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.306126 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.306157 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.306200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20240501 02:12:03.306311 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.303422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:44141 (local address 127.22.12.65:42461)
0501 02:12:03.303566 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.303576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.303634 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.303776 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.303777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.303793 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.304119 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.304122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.305059 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.305060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.306038 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.306041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.306120 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.306126 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.306157 (+ 31us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.306200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":14,"thread_start_us":67,"threads_started":1}
I20240501 02:12:03.306766 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.303487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:34911 (local address 127.22.12.66:36303)
0501 02:12:03.303536 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.303538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.303636 (+ 98us) server_negotiation.cc:407] Connection header received
0501 02:12:03.303694 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.303695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.303705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.303765 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.304188 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.304969 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.306129 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.306629 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.306637 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.306661 (+ 24us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.306693 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:12:03.306785 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.303555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:42461 (local address 127.22.12.67:44141)
0501 02:12:03.303600 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.303602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.303616 (+ 14us) server_negotiation.cc:407] Connection header received
0501 02:12:03.303700 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.303701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.303708 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.303765 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.304188 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.304985 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.306132 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.306629 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.306637 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.306668 (+ 31us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.306703 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:12:03.307083 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be92bad8fa540fdb640db9259e76438" candidate_uuid: "b94f968337694cdba51ec4b44de33808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" is_pre_election: true
I20240501 02:12:03.307088 9734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be92bad8fa540fdb640db9259e76438" candidate_uuid: "b94f968337694cdba51ec4b44de33808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5e4a74cc024e0dbd188df265940d40" is_pre_election: true
I20240501 02:12:03.307211 9734 raft_consensus.cc:2454] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94f968337694cdba51ec4b44de33808 in term 0.
I20240501 02:12:03.307217 9600 raft_consensus.cc:2454] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b94f968337694cdba51ec4b44de33808 in term 0.
I20240501 02:12:03.307432 9400 leader_election.cc:302] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5e4a74cc024e0dbd188df265940d40, b94f968337694cdba51ec4b44de33808; no voters:
I20240501 02:12:03.307564 9800 raft_consensus.cc:2781] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:03.307636 9800 raft_consensus.cc:482] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:03.307651 9800 raft_consensus.cc:3037] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.308311 9800 raft_consensus.cc:504] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.308446 9800 leader_election.cc:288] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [CANDIDATE]: Term 1 election: Requested vote from peers 3f5e4a74cc024e0dbd188df265940d40 (127.22.12.67:44141), 31f401d8f9984fcc9509b9905b5c4cf6 (127.22.12.66:36303)
I20240501 02:12:03.308595 9734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be92bad8fa540fdb640db9259e76438" candidate_uuid: "b94f968337694cdba51ec4b44de33808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5e4a74cc024e0dbd188df265940d40"
I20240501 02:12:03.308602 9600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be92bad8fa540fdb640db9259e76438" candidate_uuid: "b94f968337694cdba51ec4b44de33808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f401d8f9984fcc9509b9905b5c4cf6"
I20240501 02:12:03.308665 9734 raft_consensus.cc:3037] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.308666 9600 raft_consensus.cc:3037] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.309305 9734 raft_consensus.cc:2454] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94f968337694cdba51ec4b44de33808 in term 1.
I20240501 02:12:03.309336 9600 raft_consensus.cc:2454] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94f968337694cdba51ec4b44de33808 in term 1.
I20240501 02:12:03.309474 9400 leader_election.cc:302] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5e4a74cc024e0dbd188df265940d40, b94f968337694cdba51ec4b44de33808; no voters:
I20240501 02:12:03.309572 9800 raft_consensus.cc:2781] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:03.309752 9800 raft_consensus.cc:686] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [term 1 LEADER]: Becoming Leader. State: Replica: b94f968337694cdba51ec4b44de33808, State: Running, Role: LEADER
I20240501 02:12:03.309908 9800 consensus_queue.cc:237] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } }
I20240501 02:12:03.310647 9321 catalog_manager.cc:5484] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 reported cstate change: term changed from 0 to 1, leader changed from <none> to b94f968337694cdba51ec4b44de33808 (127.22.12.65). New cstate: current_term: 1 leader_uuid: "b94f968337694cdba51ec4b44de33808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94f968337694cdba51ec4b44de33808" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 45151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 } health_report { overall_health: UNKNOWN } } }
W20240501 02:12:03.321081 9781 tablet.cc:2369] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:12:03.331629 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.329708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.22.12.65:45151)
0501 02:12:03.329772 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.329774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.329781 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:03.329824 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.329824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.329841 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.329905 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:03.330117 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.330625 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.331203 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.331318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.331357 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:03.331449 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:03.331506 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.331534 (+ 28us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.331552 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 02:12:03.333333 9734 raft_consensus.cc:1261] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 1 FOLLOWER]: Refusing update from remote peer b94f968337694cdba51ec4b44de33808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:03.333340 9600 raft_consensus.cc:1261] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 1 FOLLOWER]: Refusing update from remote peer b94f968337694cdba51ec4b44de33808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:03.333695 9805 consensus_queue.cc:1035] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f401d8f9984fcc9509b9905b5c4cf6" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 36303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:03.333824 9800 consensus_queue.cc:1035] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f5e4a74cc024e0dbd188df265940d40" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:03.337266 9809 mvcc.cc:205] Tried to move back new op lower bound from 7022712927570636800 to 7022712927477878784. Current Snapshot: MvccSnapshot[applied={T|T < 7022712927570636800 or (T in {7022712927570636800})}]
I20240501 02:12:03.338645 9321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:03.338733 9321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:03.339444 9321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1ee1d0ffbc3b4a21b628a37039b7a169: Sending DeleteTablet for 3 replicas of tablet 7be92bad8fa540fdb640db9259e76438
I20240501 02:12:03.339776 9446 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be92bad8fa540fdb640db9259e76438 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:03 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:44884
I20240501 02:12:03.339800 9714 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be92bad8fa540fdb640db9259e76438 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:03 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:34154
I20240501 02:12:03.339834 9580 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be92bad8fa540fdb640db9259e76438 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:03 UTC) from {username='kudu', principal='kudu/127.22.12.126@KRBTEST.COM'} at 127.0.0.1:55872
I20240501 02:12:03.340034 9818 tablet_replica.cc:330] T 7be92bad8fa540fdb640db9259e76438 P b94f968337694cdba51ec4b44de33808: stopping tablet replica
I20240501 02:12:03.340152 9820 tablet_replica.cc:330] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: stopping tablet replica
I20240501 02:12:03.340237 9820 raft_consensus.cc:2229] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:03.340339 9820 raft_consensus.cc:2258] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:03.340409 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9383
I20240501 02:12:03.341346 9820 ts_tablet_manager.cc:1875] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:03.342216 9819 tablet_replica.cc:330] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: stopping tablet replica
I20240501 02:12:03.342319 9819 raft_consensus.cc:2229] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:03.342455 9819 raft_consensus.cc:2258] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:03.342916 9820 ts_tablet_manager.cc:1888] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:03.342986 9820 log.cc:1196] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/7be92bad8fa540fdb640db9259e76438
I20240501 02:12:03.342969 9819 ts_tablet_manager.cc:1875] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:03.343259 9820 ts_tablet_manager.cc:1909] T 7be92bad8fa540fdb640db9259e76438 P 31f401d8f9984fcc9509b9905b5c4cf6: Deleting consensus metadata
I20240501 02:12:03.343683 9307 catalog_manager.cc:4830] TS 31f401d8f9984fcc9509b9905b5c4cf6 (127.22.12.66:36303): tablet 7be92bad8fa540fdb640db9259e76438 (table test-table [id=b07e212e1d6d4034afd146ae3e42a872]) successfully deleted
I20240501 02:12:03.344419 9819 ts_tablet_manager.cc:1888] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:03.344491 9819 log.cc:1196] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/7be92bad8fa540fdb640db9259e76438
I20240501 02:12:03.344828 9819 ts_tablet_manager.cc:1909] T 7be92bad8fa540fdb640db9259e76438 P 3f5e4a74cc024e0dbd188df265940d40: Deleting consensus metadata
I20240501 02:12:03.345187 9306 catalog_manager.cc:4830] TS 3f5e4a74cc024e0dbd188df265940d40 (127.22.12.67:44141): tablet 7be92bad8fa540fdb640db9259e76438 (table test-table [id=b07e212e1d6d4034afd146ae3e42a872]) successfully deleted
W20240501 02:12:03.346199 9307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:45151 (error 108)
I20240501 02:12:03.346457 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9517
I20240501 02:12:03.346661 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.346510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:45151 (local address 127.0.0.1:44914)
0501 02:12:03.346583 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.346621 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:45151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20240501 02:12:03.347062 9307 catalog_manager.cc:4573] TS b94f968337694cdba51ec4b44de33808 (127.22.12.65:45151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7be92bad8fa540fdb640db9259e76438: Network error: Client connection negotiation failed: client connection to 127.22.12.65:45151: connect: Connection refused (error 111)
I20240501 02:12:03.351154 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9651
I20240501 02:12:03.355779 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9289
2024-05-01T02:12:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1917 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2024-05-01T02:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:03Z Disabled control of system clock
I20240501 02:12:03.381229 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:36923
--webserver_interface=127.22.12.126
--webserver_port=0
--builtin_ntp_servers=127.22.12.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:36923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:12:03.459160 9825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:03.459357 9825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:03.459376 9825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:03.460683 9825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:03.460723 9825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:03.460734 9825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:03.460744 9825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:03.460754 9825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:03.462205 9825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:36923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:03.462407 9825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:03.462664 9825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:03.466601 9833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.466934 9831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.467253 9830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:03.467638 9825 server_base.cc:1034] running on GCE node
I20240501 02:12:03.467850 9825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:03.468143 9825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:03.469291 9825 hybrid_clock.cc:648] HybridClock initialized: now 1714529523469269 us; error 32 us; skew 500 ppm
I20240501 02:12:03.470495 9825 webserver.cc:460] Webserver started at http://127.22.12.126:36385/ using document root <none> and password file <none>
I20240501 02:12:03.470710 9825 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:03.470754 9825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:03.470877 9825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:03.471926 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "6ac6f482f3544adcabe9a3d313a88449"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "23b8828d8eda7c1e3d4105fcee1e6ea1"
server_key_iv: "f2ec1c84669bea124031795fe8e09d82"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.472222 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "6ac6f482f3544adcabe9a3d313a88449"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "23b8828d8eda7c1e3d4105fcee1e6ea1"
server_key_iv: "f2ec1c84669bea124031795fe8e09d82"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.473392 9825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:12:03.474077 9839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.474225 9825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:03.474313 9825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "6ac6f482f3544adcabe9a3d313a88449"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "23b8828d8eda7c1e3d4105fcee1e6ea1"
server_key_iv: "f2ec1c84669bea124031795fe8e09d82"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.474387 9825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:03.486131 9825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:03.486421 9825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:03.486555 9825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:03.490082 9825 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:36923
I20240501 02:12:03.490113 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:36923 every 8 connection(s)
I20240501 02:12:03.490413 9825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:03.491031 9892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.493507 9892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449: Bootstrap starting.
I20240501 02:12:03.494056 9892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.494299 9892 log.cc:824] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.494791 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9825
I20240501 02:12:03.494901 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:03.495070 22577 external_mini_cluster.cc:1428] Setting key 0992a8a7a4f05634176b2fd6c434448b
I20240501 02:12:03.495082 9892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449: No bootstrap required, opened a new log
I20240501 02:12:03.497011 9892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6f482f3544adcabe9a3d313a88449" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36923 } }
I20240501 02:12:03.497138 9892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.497155 9892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac6f482f3544adcabe9a3d313a88449, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.497243 9892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6f482f3544adcabe9a3d313a88449" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36923 } }
I20240501 02:12:03.497304 9892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:03.497330 9892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:03.497377 9892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.498317 9892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6f482f3544adcabe9a3d313a88449" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36923 } }
I20240501 02:12:03.498438 9892 leader_election.cc:302] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ac6f482f3544adcabe9a3d313a88449; no voters:
I20240501 02:12:03.498615 9892 leader_election.cc:288] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:03.498667 9897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:03.498829 9897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [term 1 LEADER]: Becoming Leader. State: Replica: 6ac6f482f3544adcabe9a3d313a88449, State: Running, Role: LEADER
I20240501 02:12:03.498855 9892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:03.498955 9897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6f482f3544adcabe9a3d313a88449" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36923 } }
I20240501 02:12:03.499300 9899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ac6f482f3544adcabe9a3d313a88449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6f482f3544adcabe9a3d313a88449" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36923 } } }
I20240501 02:12:03.499398 9899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:03.499586 9904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:03.500053 9904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:03.500265 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.495612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.22.12.126:36923)
0501 02:12:03.495795 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.495801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.495819 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:12:03.496223 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.496230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.496281 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.496412 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.496832 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.497720 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.498520 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.499366 (+ 846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.499602 (+ 236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.499612 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.499612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:03.499781 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.499804 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:03.499812 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.499889 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.499982 (+ 93us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.500035 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20240501 02:12:03.500272 9898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac6f482f3544adcabe9a3d313a88449. Latest consensus state: current_term: 1 leader_uuid: "6ac6f482f3544adcabe9a3d313a88449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac6f482f3544adcabe9a3d313a88449" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 36923 } } }
I20240501 02:12:03.500609 9898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:03.501868 9904 catalog_manager.cc:1347] Generated new cluster ID: 1b763fd5ef66430db51715733a9a9c1c
I20240501 02:12:03.501919 9904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:03.509510 9904 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:03.510071 9904 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:03.524981 9904 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449: Generated new TSK 0
I20240501 02:12:03.525198 9904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:03.526479 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.65:0
--local_ip_for_outbound_sockets=127.22.12.65
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36923
--builtin_ntp_servers=127.22.12.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:12:03.624349 9916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:03.624536 9916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:03.624567 9916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:03.625912 9916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:03.625958 9916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:03.625990 9916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.65
I20240501 02:12:03.627493 9916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.12.65
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.65
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:03.627725 9916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:03.627964 9916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:03.632597 9921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.632879 9922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.633260 9924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:03.633571 9916 server_base.cc:1034] running on GCE node
I20240501 02:12:03.633754 9916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:03.633965 9916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:03.635097 9916 hybrid_clock.cc:648] HybridClock initialized: now 1714529523635084 us; error 30 us; skew 500 ppm
I20240501 02:12:03.636372 9916 webserver.cc:460] Webserver started at http://127.22.12.65:34047/ using document root <none> and password file <none>
I20240501 02:12:03.636557 9916 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:03.636600 9916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:03.636711 9916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:03.637703 9916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data/instance:
uuid: "c25a930370434b1394136b2089dd064f"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "08242f3644a034f1918ce5456bce2a76"
server_key_iv: "d3495a1cdd5e7417944a1133aa9e67a2"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.637997 9916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance:
uuid: "c25a930370434b1394136b2089dd064f"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "08242f3644a034f1918ce5456bce2a76"
server_key_iv: "d3495a1cdd5e7417944a1133aa9e67a2"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.639127 9916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240501 02:12:03.639770 9930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.639932 9916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:03.640008 9916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal
uuid: "c25a930370434b1394136b2089dd064f"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "08242f3644a034f1918ce5456bce2a76"
server_key_iv: "d3495a1cdd5e7417944a1133aa9e67a2"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.640074 9916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:03.652354 9916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:03.652618 9916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:03.652711 9916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:03.652909 9916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:03.653237 9916 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:03.653273 9916 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.653295 9916 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:03.653306 9916 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.659654 9916 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.65:37669
I20240501 02:12:03.659695 10043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.65:37669 every 8 connection(s)
I20240501 02:12:03.659988 9916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/data/info.pb
I20240501 02:12:03.661605 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 9916
I20240501 02:12:03.661711 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-0/wal/instance
I20240501 02:12:03.661882 22577 external_mini_cluster.cc:1428] Setting key 220e051c6e8a1edbbba6cf6f41e4005c
I20240501 02:12:03.662881 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.66:0
--local_ip_for_outbound_sockets=127.22.12.66
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36923
--builtin_ntp_servers=127.22.12.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 02:12:03.665158 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.660696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:49175 (local address 127.22.12.126:36923)
0501 02:12:03.660764 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.660766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.661004 (+ 238us) server_negotiation.cc:407] Connection header received
0501 02:12:03.661656 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.661657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.661670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.661729 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.662358 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.662970 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.663537 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.663702 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.664846 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.664850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.664851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:03.664856 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.664875 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:03.664880 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.664940 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.665038 (+ 98us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.665075 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:12:03.665321 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.660638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36923 (local address 127.22.12.65:49175)
0501 02:12:03.660885 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.660916 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.661528 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.661823 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.661828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.661839 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:03.662268 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.662273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.663083 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.663084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.663458 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.663460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.663593 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.664687 (+ 1094us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:12:03.664688 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:03.664716 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:12:03.664741 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:03.664950 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:03.664954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:03.664957 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.665025 (+ 68us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.665101 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20240501 02:12:03.665721 10044 heartbeater.cc:347] Connected to a master server at 127.22.12.126:36923
I20240501 02:12:03.665819 10044 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:03.666021 10044 heartbeater.cc:511] Master 127.22.12.126:36923 requested a full tablet report, sending...
I20240501 02:12:03.666481 9856 ts_manager.cc:194] Registered new tserver with Master: c25a930370434b1394136b2089dd064f (127.22.12.65:37669)
I20240501 02:12:03.667114 9856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.65:49175
W20240501 02:12:03.743450 10047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:03.743640 10047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:03.743664 10047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:03.745057 10047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:03.745105 10047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:03.745131 10047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.66
I20240501 02:12:03.746711 10047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.12.66
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.66
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:03.746927 10047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:03.747138 10047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:03.751509 10053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.751614 10052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.751942 10055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:03.752441 10047 server_base.cc:1034] running on GCE node
I20240501 02:12:03.752875 10047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:03.753463 10047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:03.754590 10047 hybrid_clock.cc:648] HybridClock initialized: now 1714529523754573 us; error 33 us; skew 500 ppm
I20240501 02:12:03.755813 10047 webserver.cc:460] Webserver started at http://127.22.12.66:39857/ using document root <none> and password file <none>
I20240501 02:12:03.755997 10047 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:03.756038 10047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:03.756141 10047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:03.757092 10047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data/instance:
uuid: "7bda4242e95e4b9da34a6cbb20b06551"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "b0382de6b4af090e1c367498f908599d"
server_key_iv: "b134c361fd02e10d9373458144593053"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.757393 10047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance:
uuid: "7bda4242e95e4b9da34a6cbb20b06551"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "b0382de6b4af090e1c367498f908599d"
server_key_iv: "b134c361fd02e10d9373458144593053"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.758488 10047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:03.759135 10061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.759287 10047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:03.759361 10047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal
uuid: "7bda4242e95e4b9da34a6cbb20b06551"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "b0382de6b4af090e1c367498f908599d"
server_key_iv: "b134c361fd02e10d9373458144593053"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.759414 10047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:03.775519 10047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:03.775815 10047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:03.775916 10047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:03.776113 10047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:03.776432 10047 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:03.776472 10047 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.776497 10047 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:03.776512 10047 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.781723 10047 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.66:38495
I20240501 02:12:03.781769 10174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.66:38495 every 8 connection(s)
I20240501 02:12:03.782058 10047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/data/info.pb
I20240501 02:12:03.786414 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.782738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.66:38267 (local address 127.22.12.126:36923)
0501 02:12:03.782785 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.782787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.782996 (+ 209us) server_negotiation.cc:407] Connection header received
0501 02:12:03.783498 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.783500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.783512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.783564 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.784023 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.784500 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.785043 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.785200 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.786138 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.786144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.786144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:03.786149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.786164 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:03.786167 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.786209 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.786295 (+ 86us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.786331 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:12:03.786574 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.782723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36923 (local address 127.22.12.66:38267)
0501 02:12:03.782891 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.782915 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.783390 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.783623 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.783627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.783635 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:03.783936 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.783939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.784581 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.784582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.784960 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.784962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.785097 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.786002 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:12:03.786003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:03.786023 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:12:03.786043 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:03.786219 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:03.786223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:03.786226 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.786293 (+ 67us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.786375 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20240501 02:12:03.787046 10175 heartbeater.cc:347] Connected to a master server at 127.22.12.126:36923
I20240501 02:12:03.787139 10175 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:03.787329 10175 heartbeater.cc:511] Master 127.22.12.126:36923 requested a full tablet report, sending...
I20240501 02:12:03.787678 9856 ts_manager.cc:194] Registered new tserver with Master: 7bda4242e95e4b9da34a6cbb20b06551 (127.22.12.66:38495)
I20240501 02:12:03.787889 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10047
I20240501 02:12:03.787992 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal/instance
I20240501 02:12:03.788168 22577 external_mini_cluster.cc:1428] Setting key 9a1207cc9e852324361c5eb2d32273b7
I20240501 02:12:03.788185 9856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.66:38267
I20240501 02:12:03.789177 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.12.67:0
--local_ip_for_outbound_sockets=127.22.12.67
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36923
--builtin_ntp_servers=127.22.12.84:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 02:12:03.866361 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:03.866554 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:03.866573 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:03.867825 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:03.867863 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:03.867885 10178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.12.67
I20240501 02:12:03.869256 10178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.12.67
--webserver_port=0
--tserver_master_addrs=127.22.12.126:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.12.67
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:03.869442 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:03.869653 10178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:03.873783 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.873956 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:03.874395 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:03.874718 10178 server_base.cc:1034] running on GCE node
I20240501 02:12:03.874907 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:03.875123 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:03.876258 10178 hybrid_clock.cc:648] HybridClock initialized: now 1714529523876242 us; error 34 us; skew 500 ppm
I20240501 02:12:03.877507 10178 webserver.cc:460] Webserver started at http://127.22.12.67:46249/ using document root <none> and password file <none>
I20240501 02:12:03.877691 10178 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:03.877733 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:03.877854 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:03.878911 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data/instance:
uuid: "ef3847a80da947e4821827cb9fdfca56"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "5f0e763e66f0641071e2c9685fb385dd"
server_key_iv: "cc564a4a60eb2b25ea12c4c7437eb5d7"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.879201 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance:
uuid: "ef3847a80da947e4821827cb9fdfca56"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "5f0e763e66f0641071e2c9685fb385dd"
server_key_iv: "cc564a4a60eb2b25ea12c4c7437eb5d7"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.880309 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:03.880937 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.881088 10178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:03.881170 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal
uuid: "ef3847a80da947e4821827cb9fdfca56"
format_stamp: "Formatted at 2024-05-01 02:12:03 on dist-test-slave-9tkh"
server_key: "5f0e763e66f0641071e2c9685fb385dd"
server_key_iv: "cc564a4a60eb2b25ea12c4c7437eb5d7"
server_key_version: "encryptionkey@0"
I20240501 02:12:03.881227 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:03.897873 10178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:03.898166 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:03.898315 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:03.898541 10178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 02:12:03.898865 10178 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 02:12:03.898896 10178 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.898926 10178 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 02:12:03.898942 10178 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:03.904194 10178 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.67:42113
I20240501 02:12:03.904243 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.67:42113 every 8 connection(s)
I20240501 02:12:03.904546 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/data/info.pb
I20240501 02:12:03.909044 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.905261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.67:33207 (local address 127.22.12.126:36923)
0501 02:12:03.905315 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.905317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.905508 (+ 191us) server_negotiation.cc:407] Connection header received
0501 02:12:03.906003 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.906004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.906017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.906061 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.906616 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.907115 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.907656 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.907820 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.908765 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.908771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.908771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:03.908776 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.908791 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:03.908795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.908836 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.908925 (+ 89us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.908961 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:12:03.909169 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.905219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:36923 (local address 127.22.12.67:33207)
0501 02:12:03.905405 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.905430 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.905898 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.906133 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.906138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.906165 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:03.906489 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.906492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.907201 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.907202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.907571 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.907573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.907710 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.908621 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 02:12:03.908621 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:03.908642 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 02:12:03.908663 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:03.908848 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:03.908853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:03.908856 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.908924 (+ 68us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.908997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20240501 02:12:03.909651 10306 heartbeater.cc:347] Connected to a master server at 127.22.12.126:36923
I20240501 02:12:03.909739 10306 heartbeater.cc:464] Registering TS with master...
I20240501 02:12:03.909931 10306 heartbeater.cc:511] Master 127.22.12.126:36923 requested a full tablet report, sending...
I20240501 02:12:03.910321 9856 ts_manager.cc:194] Registered new tserver with Master: ef3847a80da947e4821827cb9fdfca56 (127.22.12.67:42113)
I20240501 02:12:03.910831 9856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.12.67:33207
I20240501 02:12:03.913744 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10178
I20240501 02:12:03.913832 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal/instance
I20240501 02:12:03.913987 22577 external_mini_cluster.cc:1428] Setting key 75245c144cda4e3a5bc8e3427599aff7
I20240501 02:12:03.914945 22577 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 02:12:03.918459 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.916207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.22.12.126:36923)
0501 02:12:03.916255 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.916257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.916344 (+ 87us) server_negotiation.cc:407] Connection header received
0501 02:12:03.916393 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.916394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.916403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.916438 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:03.916736 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.917198 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.917773 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.917871 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.918274 (+ 403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:03.918278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:03.918278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:03.918281 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:03.918295 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:03.918298 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:03.918336 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.918377 (+ 41us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.918402 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:12:03.919945 9856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 02:12:03.921465 9856 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 02:12:03.926466 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.923279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:42113 (local address 127.0.0.1:55242)
0501 02:12:03.923421 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.923449 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.923520 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.924109 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.924110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.924111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.924284 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.924285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.925303 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.925303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.926151 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.926153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.926239 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.926281 (+ 42us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.926334 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.926378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20240501 02:12:03.926843 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.923279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37669 (local address 127.0.0.1:56026)
0501 02:12:03.923618 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.923626 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.923663 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.923839 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.923842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.923844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.924072 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.924075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.925470 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.925471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.926258 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.926260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.926316 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.926318 (+ 2us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.926346 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.926378 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20240501 02:12:03.927039 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.923363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.22.12.67:42113)
0501 02:12:03.923963 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.923971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.923979 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:03.924012 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.924014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.924028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.924103 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.924356 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.925221 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.926250 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.926790 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.926881 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.926919 (+ 38us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.926960 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":84,"threads_started":1}
I20240501 02:12:03.927398 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.923509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:38495 (local address 127.0.0.1:47804)
0501 02:12:03.923633 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.923642 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.923691 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.923910 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.923911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.923913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.924088 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.924090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.926410 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.926411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.927214 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.927217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.927274 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.927277 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.927303 (+ 26us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.927336 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20240501 02:12:03.927603 10240 tablet_service.cc:1467] Processing CreateTablet for tablet 83eb669c467d45a981095e0ffbe01a03 (DEFAULT_TABLE table=test-table [id=d100d464b77f4bcdb97095ea07f07783]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:03.927711 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.923479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.22.12.65:37669)
0501 02:12:03.923653 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.923663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.923674 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:12:03.923714 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.923716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.923733 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.923822 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.924158 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.925396 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.926325 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.927363 (+ 1038us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.927569 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.927603 (+ 34us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.927644 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20240501 02:12:03.927907 10240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83eb669c467d45a981095e0ffbe01a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.928002 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.923612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.22.12.66:38495)
0501 02:12:03.923777 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.923783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.923790 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:03.923819 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.923820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.923832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.923897 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.924248 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.925404 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.927292 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.927790 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.927868 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.927898 (+ 30us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.927930 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20240501 02:12:03.928496 10109 tablet_service.cc:1467] Processing CreateTablet for tablet 83eb669c467d45a981095e0ffbe01a03 (DEFAULT_TABLE table=test-table [id=d100d464b77f4bcdb97095ea07f07783]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:03.928741 10109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83eb669c467d45a981095e0ffbe01a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.930493 10320 tablet_bootstrap.cc:492] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Bootstrap starting.
I20240501 02:12:03.930757 9978 tablet_service.cc:1467] Processing CreateTablet for tablet 83eb669c467d45a981095e0ffbe01a03 (DEFAULT_TABLE table=test-table [id=d100d464b77f4bcdb97095ea07f07783]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 02:12:03.930991 9978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83eb669c467d45a981095e0ffbe01a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:03.931146 10320 tablet_bootstrap.cc:654] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.931327 10321 tablet_bootstrap.cc:492] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Bootstrap starting.
I20240501 02:12:03.931423 10320 log.cc:824] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.931908 10321 tablet_bootstrap.cc:654] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.932049 10320 tablet_bootstrap.cc:492] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: No bootstrap required, opened a new log
I20240501 02:12:03.932142 10320 ts_tablet_manager.cc:1367] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:03.932190 10321 log.cc:824] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.933494 10324 tablet_bootstrap.cc:492] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: Bootstrap starting.
I20240501 02:12:03.934047 10324 tablet_bootstrap.cc:654] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:03.934079 10320 raft_consensus.cc:348] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.934229 10320 raft_consensus.cc:374] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.934258 10320 raft_consensus.cc:729] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3847a80da947e4821827cb9fdfca56, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.934322 10324 log.cc:824] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:03.934358 10320 consensus_queue.cc:260] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.934623 10320 ts_tablet_manager.cc:1398] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 02:12:03.934648 10306 heartbeater.cc:503] Master 127.22.12.126:36923 was elected leader, sending a full tablet report...
I20240501 02:12:03.935153 10321 tablet_bootstrap.cc:492] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: No bootstrap required, opened a new log
I20240501 02:12:03.935231 10321 ts_tablet_manager.cc:1367] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 02:12:03.935279 10324 tablet_bootstrap.cc:492] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: No bootstrap required, opened a new log
I20240501 02:12:03.935339 10324 ts_tablet_manager.cc:1367] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 02:12:03.937422 10324 raft_consensus.cc:348] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.937422 10321 raft_consensus.cc:348] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.937566 10324 raft_consensus.cc:374] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.937566 10321 raft_consensus.cc:374] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:03.937590 10324 raft_consensus.cc:729] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25a930370434b1394136b2089dd064f, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.937590 10321 raft_consensus.cc:729] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bda4242e95e4b9da34a6cbb20b06551, State: Initialized, Role: FOLLOWER
I20240501 02:12:03.937682 10324 consensus_queue.cc:260] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.937682 10321 consensus_queue.cc:260] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.937932 10321 ts_tablet_manager.cc:1398] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:12:03.937932 10324 ts_tablet_manager.cc:1398] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 02:12:03.937990 10044 heartbeater.cc:503] Master 127.22.12.126:36923 was elected leader, sending a full tablet report...
I20240501 02:12:03.938038 10175 heartbeater.cc:503] Master 127.22.12.126:36923 was elected leader, sending a full tablet report...
I20240501 02:12:03.981149 10327 raft_consensus.cc:482] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 02:12:03.981300 10327 raft_consensus.cc:504] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.981667 10327 leader_election.cc:288] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef3847a80da947e4821827cb9fdfca56 (127.22.12.67:42113), 7bda4242e95e4b9da34a6cbb20b06551 (127.22.12.66:38495)
I20240501 02:12:03.983770 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.981839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.67:42113 (local address 127.22.12.65:43177)
0501 02:12:03.981894 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.981906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.981948 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.982085 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.982087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.982089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.982309 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.982311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.983002 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.983003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.983557 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.983558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.983623 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.983627 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.983654 (+ 27us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.983694 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20240501 02:12:03.983907 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.981935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.66:38495 (local address 127.22.12.65:34265)
0501 02:12:03.982102 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:03.982110 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:03.982152 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:03.982252 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:03.982253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:03.982254 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 02:12:03.982443 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.982444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.983089 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.983090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:03.983654 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:03.983655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.983730 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.983734 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 02:12:03.983763 (+ 29us) client_negotiation.cc:240] Negotiation successful
0501 02:12:03.983808 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20240501 02:12:03.984246 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.981898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:43177 (local address 127.22.12.67:42113)
0501 02:12:03.981949 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.981951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.981958 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:03.982004 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.982005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.982015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.982066 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.982372 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.982915 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.983638 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.984124 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.984133 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.984172 (+ 39us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.984198 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 02:12:03.984265 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.982100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.12.65:34265 (local address 127.22.12.66:38495)
0501 02:12:03.982149 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.982151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.982159 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:03.982190 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.982191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.982202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.982252 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 02:12:03.982516 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.983015 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.983736 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.984161 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:03.984169 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:03.984191 (+ 22us) server_negotiation.cc:299] Negotiation successful
0501 02:12:03.984220 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:12:03.984555 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83eb669c467d45a981095e0ffbe01a03" candidate_uuid: "c25a930370434b1394136b2089dd064f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3847a80da947e4821827cb9fdfca56" is_pre_election: true
I20240501 02:12:03.984545 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83eb669c467d45a981095e0ffbe01a03" candidate_uuid: "c25a930370434b1394136b2089dd064f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bda4242e95e4b9da34a6cbb20b06551" is_pre_election: true
I20240501 02:12:03.984692 10129 raft_consensus.cc:2454] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25a930370434b1394136b2089dd064f in term 0.
I20240501 02:12:03.984694 10260 raft_consensus.cc:2454] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25a930370434b1394136b2089dd064f in term 0.
I20240501 02:12:03.984934 9932 leader_election.cc:302] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25a930370434b1394136b2089dd064f, ef3847a80da947e4821827cb9fdfca56; no voters:
I20240501 02:12:03.985074 10327 raft_consensus.cc:2781] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 02:12:03.985152 10327 raft_consensus.cc:482] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 02:12:03.985183 10327 raft_consensus.cc:3037] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.985841 10327 raft_consensus.cc:504] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.985962 10327 leader_election.cc:288] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [CANDIDATE]: Term 1 election: Requested vote from peers ef3847a80da947e4821827cb9fdfca56 (127.22.12.67:42113), 7bda4242e95e4b9da34a6cbb20b06551 (127.22.12.66:38495)
I20240501 02:12:03.986141 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83eb669c467d45a981095e0ffbe01a03" candidate_uuid: "c25a930370434b1394136b2089dd064f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3847a80da947e4821827cb9fdfca56"
I20240501 02:12:03.986166 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83eb669c467d45a981095e0ffbe01a03" candidate_uuid: "c25a930370434b1394136b2089dd064f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bda4242e95e4b9da34a6cbb20b06551"
I20240501 02:12:03.986227 10260 raft_consensus.cc:3037] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.986227 10129 raft_consensus.cc:3037] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:03.987112 10129 raft_consensus.cc:2454] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25a930370434b1394136b2089dd064f in term 1.
I20240501 02:12:03.987112 10260 raft_consensus.cc:2454] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25a930370434b1394136b2089dd064f in term 1.
I20240501 02:12:03.987283 9932 leader_election.cc:302] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bda4242e95e4b9da34a6cbb20b06551, c25a930370434b1394136b2089dd064f; no voters:
I20240501 02:12:03.987371 10327 raft_consensus.cc:2781] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:03.987551 10327 raft_consensus.cc:686] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 1 LEADER]: Becoming Leader. State: Replica: c25a930370434b1394136b2089dd064f, State: Running, Role: LEADER
I20240501 02:12:03.987653 10327 consensus_queue.cc:237] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } }
I20240501 02:12:03.988438 9856 catalog_manager.cc:5484] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f reported cstate change: term changed from 0 to 1, leader changed from <none> to c25a930370434b1394136b2089dd064f (127.22.12.65). New cstate: current_term: 1 leader_uuid: "c25a930370434b1394136b2089dd064f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25a930370434b1394136b2089dd064f" member_type: VOTER last_known_addr { host: "127.22.12.65" port: 37669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 } health_report { overall_health: UNKNOWN } } }
I20240501 02:12:04.000535 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:03.998374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.22.12.65:37669)
0501 02:12:03.998422 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:03.998424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:03.998431 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:03.998503 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:03.998504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:03.998551 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:03.998627 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:03.998851 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:03.999394 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:03.999983 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:04.000112 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:04.000159 (+ 47us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:04.000284 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:04.000345 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:04.000436 (+ 91us) server_negotiation.cc:299] Negotiation successful
0501 02:12:04.000462 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240501 02:12:04.002151 10129 raft_consensus.cc:1261] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 1 FOLLOWER]: Refusing update from remote peer c25a930370434b1394136b2089dd064f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:04.002158 10260 raft_consensus.cc:1261] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 1 FOLLOWER]: Refusing update from remote peer c25a930370434b1394136b2089dd064f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:04.002480 10327 consensus_queue.cc:1035] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bda4242e95e4b9da34a6cbb20b06551" member_type: VOTER last_known_addr { host: "127.22.12.66" port: 38495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:04.002651 10330 consensus_queue.cc:1035] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3847a80da947e4821827cb9fdfca56" member_type: VOTER last_known_addr { host: "127.22.12.67" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:04.004933 10333 mvcc.cc:205] Tried to move back new op lower bound from 7022712930310475776 to 7022712930254102528. Current Snapshot: MvccSnapshot[applied={T|T < 7022712930310475776}]
I20240501 02:12:04.007287 9856 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40662:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:04.007459 9856 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40662:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 02:12:04.008231 9856 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6ac6f482f3544adcabe9a3d313a88449: Sending DeleteTablet for 3 replicas of tablet 83eb669c467d45a981095e0ffbe01a03
I20240501 02:12:04.008550 10109 tablet_service.cc:1514] Processing DeleteTablet for tablet 83eb669c467d45a981095e0ffbe01a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:04 UTC) from {username='slave'} at 127.0.0.1:47804
I20240501 02:12:04.008548 9978 tablet_service.cc:1514] Processing DeleteTablet for tablet 83eb669c467d45a981095e0ffbe01a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:04 UTC) from {username='slave'} at 127.0.0.1:56026
I20240501 02:12:04.008582 10240 tablet_service.cc:1514] Processing DeleteTablet for tablet 83eb669c467d45a981095e0ffbe01a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 02:12:04 UTC) from {username='slave'} at 127.0.0.1:55242
I20240501 02:12:04.008780 10344 tablet_replica.cc:330] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: stopping tablet replica
I20240501 02:12:04.008865 10344 raft_consensus.cc:2229] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:04.008913 10345 tablet_replica.cc:330] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f: stopping tablet replica
I20240501 02:12:04.008951 10344 raft_consensus.cc:2258] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:04.008977 10345 raft_consensus.cc:2229] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 1 LEADER]: Raft consensus shutting down.
I20240501 02:12:04.009027 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9916
I20240501 02:12:04.009052 10345 raft_consensus.cc:2258] T 83eb669c467d45a981095e0ffbe01a03 P c25a930370434b1394136b2089dd064f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:04.009346 10344 ts_tablet_manager.cc:1875] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:04.009742 10343 tablet_replica.cc:330] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: stopping tablet replica
I20240501 02:12:04.009830 10343 raft_consensus.cc:2229] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 02:12:04.009917 10343 raft_consensus.cc:2258] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 02:12:04.010301 10343 ts_tablet_manager.cc:1875] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 02:12:04.011112 10344 ts_tablet_manager.cc:1888] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:04.011185 10344 log.cc:1196] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-1/wal/wals/83eb669c467d45a981095e0ffbe01a03
I20240501 02:12:04.011446 10344 ts_tablet_manager.cc:1909] T 83eb669c467d45a981095e0ffbe01a03 P 7bda4242e95e4b9da34a6cbb20b06551: Deleting consensus metadata
I20240501 02:12:04.011698 10343 ts_tablet_manager.cc:1888] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 02:12:04.011775 10343 log.cc:1196] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Deleting WAL directory at /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714529249419714-22577-0/minicluster-data/ts-2/wal/wals/83eb669c467d45a981095e0ffbe01a03
I20240501 02:12:04.011816 9841 catalog_manager.cc:4830] TS 7bda4242e95e4b9da34a6cbb20b06551 (127.22.12.66:38495): tablet 83eb669c467d45a981095e0ffbe01a03 (table test-table [id=d100d464b77f4bcdb97095ea07f07783]) successfully deleted
I20240501 02:12:04.012022 10343 ts_tablet_manager.cc:1909] T 83eb669c467d45a981095e0ffbe01a03 P ef3847a80da947e4821827cb9fdfca56: Deleting consensus metadata
I20240501 02:12:04.012360 9841 catalog_manager.cc:4830] TS ef3847a80da947e4821827cb9fdfca56 (127.22.12.67:42113): tablet 83eb669c467d45a981095e0ffbe01a03 (table test-table [id=d100d464b77f4bcdb97095ea07f07783]) successfully deleted
W20240501 02:12:04.014812 9842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.12.65:37669 (error 108)
I20240501 02:12:04.015260 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:04.015138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.65:37669 (local address 127.0.0.1:56048)
0501 02:12:04.015189 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:04.015226 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20240501 02:12:04.015311 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10047
W20240501 02:12:04.015376 9842 catalog_manager.cc:4573] TS c25a930370434b1394136b2089dd064f (127.22.12.65:37669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83eb669c467d45a981095e0ffbe01a03: Network error: Client connection negotiation failed: client connection to 127.22.12.65:37669: connect: Connection refused (error 111)
I20240501 02:12:04.019909 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10178
I20240501 02:12:04.023802 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 9825
2024-05-01T02:12:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (667 ms)
[----------] 6 tests from EncryptionPolicyTest (7824 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:04 dist-test-slave-9tkh krb5kdc[10349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:04 dist-test-slave-9tkh krb5kdc[10349](info): set up 2 sockets
May 01 02:12:04 dist-test-slave-9tkh krb5kdc[10349](info): commencing operation
krb5kdc: starting...
W20240501 02:12:04.910297 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:04 dist-test-slave-9tkh krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:05.020982 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:05.099346 10362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:05.099567 10362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:05.099599 10362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:05.100931 10362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:05.100983 10362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:05.101001 10362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:05.101016 10362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:05.101030 10362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:05.102758 10362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:05.102993 10362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:05.103178 10362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:05.107180 10367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:05.107445 10370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:05.107815 10368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:05.108237 10362 server_base.cc:1034] running on GCE node
I20240501 02:12:05.108453 10362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:05.108698 10362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:05.109834 10362 hybrid_clock.cc:648] HybridClock initialized: now 1714529525109816 us; error 33 us; skew 500 ppm
May 01 02:12:05 dist-test-slave-9tkh krb5kdc[10349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:05.112332 10362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:05.112748 10362 webserver.cc:460] Webserver started at http://127.0.0.1:46493/ using document root <none> and password file <none>
I20240501 02:12:05.112927 10362 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:05.112969 10362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:05.113076 10362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:05.114090 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "4fff565e25ec456791d1e199c4fdd4f3"
format_stamp: "Formatted at 2024-05-01 02:12:05 on dist-test-slave-9tkh"
server_key: "2e138ef420d69053138c68ba75a1a859"
server_key_iv: "d3e9dc2cf5ba85a41f727f4c2b2e0ed2"
server_key_version: "encryptionkey@0"
I20240501 02:12:05.114391 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "4fff565e25ec456791d1e199c4fdd4f3"
format_stamp: "Formatted at 2024-05-01 02:12:05 on dist-test-slave-9tkh"
server_key: "2e138ef420d69053138c68ba75a1a859"
server_key_iv: "d3e9dc2cf5ba85a41f727f4c2b2e0ed2"
server_key_version: "encryptionkey@0"
I20240501 02:12:05.115653 10362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:05.116375 10377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:05.116544 10362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:05.116626 10362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "4fff565e25ec456791d1e199c4fdd4f3"
format_stamp: "Formatted at 2024-05-01 02:12:05 on dist-test-slave-9tkh"
server_key: "2e138ef420d69053138c68ba75a1a859"
server_key_iv: "d3e9dc2cf5ba85a41f727f4c2b2e0ed2"
server_key_version: "encryptionkey@0"
I20240501 02:12:05.116688 10362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:05.132598 10362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:05.134817 10362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:05.134969 10362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:05.138641 10362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45595
I20240501 02:12:05.138684 10429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45595 every 8 connection(s)
I20240501 02:12:05.139041 10362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:05.139865 10430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:05.142079 10430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3: Bootstrap starting.
I20240501 02:12:05.142683 10430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:05.142931 10430 log.cc:824] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:05.143676 10430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3: No bootstrap required, opened a new log
I20240501 02:12:05.145113 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10362
I20240501 02:12:05.145210 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:05.145169 10430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fff565e25ec456791d1e199c4fdd4f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20240501 02:12:05.145328 10430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:05.145356 10430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fff565e25ec456791d1e199c4fdd4f3, State: Initialized, Role: FOLLOWER
I20240501 02:12:05.145385 22577 external_mini_cluster.cc:1428] Setting key 0439a4de0afcba7939a642905f8b8273
I20240501 02:12:05.145493 10430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fff565e25ec456791d1e199c4fdd4f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20240501 02:12:05.145573 10430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:05.145604 10430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:05.145651 10430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:05.146306 10430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fff565e25ec456791d1e199c4fdd4f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20240501 02:12:05.146407 10430 leader_election.cc:302] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fff565e25ec456791d1e199c4fdd4f3; no voters:
I20240501 02:12:05.146572 10430 leader_election.cc:288] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:05.146770 10435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:05.146837 10430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:05.146929 10435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4fff565e25ec456791d1e199c4fdd4f3, State: Running, Role: LEADER
I20240501 02:12:05.147024 10435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fff565e25ec456791d1e199c4fdd4f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } }
I20240501 02:12:05.147368 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fff565e25ec456791d1e199c4fdd4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fff565e25ec456791d1e199c4fdd4f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } } }
I20240501 02:12:05.147457 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:05.147617 10437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fff565e25ec456791d1e199c4fdd4f3. Latest consensus state: current_term: 1 leader_uuid: "4fff565e25ec456791d1e199c4fdd4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fff565e25ec456791d1e199c4fdd4f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45595 } } }
I20240501 02:12:05.147670 10437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:05.147768 10440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:05.148483 10440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:05.150689 10440 catalog_manager.cc:1347] Generated new cluster ID: 5f9179496c6047fa9244028eb102b3e4
I20240501 02:12:05.150748 10440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:12:05 dist-test-slave-9tkh krb5kdc[10349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 02:12:05.156085 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:05.145852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49161 (local address 127.0.0.1:45595)
0501 02:12:05.146057 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:05.146070 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:05.146094 (+ 24us) server_negotiation.cc:407] Connection header received
0501 02:12:05.146190 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:05.146195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:05.146234 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:05.146366 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:05.147427 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:05.148669 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:05.149496 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:05.149758 (+ 262us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:05.152350 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:05.152373 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:05.152373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:05.152396 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:05.154454 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:05.154912 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:05.154916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:05.154916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:05.154989 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:05.155174 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:05.155174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:05.155175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:05.155432 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:05.155575 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:05.155731 (+ 156us) server_negotiation.cc:299] Negotiation successful
0501 02:12:05.155859 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20240501 02:12:05.157105 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:05.165284 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:05.158576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48005 (local address 127.0.0.1:45595)
0501 02:12:05.158613 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:05.158615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:05.158624 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:12:05.158693 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:05.158694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:05.158704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:05.158757 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:05.159479 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:05.160223 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:05.160897 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:05.161087 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:05.162144 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:05.162148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:05.162148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:05.162154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:05.163817 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:05.164216 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:05.164217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:05.164218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:05.164242 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:05.164406 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:05.164407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:05.164407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:05.164515 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:05.164645 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:05.164769 (+ 124us) server_negotiation.cc:299] Negotiation successful
0501 02:12:05.165176 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:05.168470 10440 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:05.168989 10440 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:05.172329 10440 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4fff565e25ec456791d1e199c4fdd4f3: Generated new TSK 0
I20240501 02:12:05.172508 10440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:05.178681 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10362
W20240501 02:12:05.182984 10347 connection.cc:535] client connection to 127.0.0.1:45595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45595): Connection reset by peer (error 104)
2024-05-01T02:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:05 dist-test-slave-9tkh krb5kdc[10461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:05 dist-test-slave-9tkh krb5kdc[10461](info): set up 2 sockets
May 01 02:12:05 dist-test-slave-9tkh krb5kdc[10461](info): commencing operation
krb5kdc: starting...
W20240501 02:12:06.078230 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:06 dist-test-slave-9tkh krb5kdc[10461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:06.190954 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:06.269369 10474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:06.269572 10474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:06.269593 10474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:06.270933 10474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:06.270982 10474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:06.270993 10474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:06.271004 10474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:06.271014 10474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:06.272670 10474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:06.272871 10474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:06.273026 10474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:06.277123 10479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:06.277433 10480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:06.277606 10482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:06.277982 10474 server_base.cc:1034] running on GCE node
I20240501 02:12:06.278184 10474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:06.278412 10474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:06.279542 10474 hybrid_clock.cc:648] HybridClock initialized: now 1714529526279520 us; error 32 us; skew 500 ppm
May 01 02:12:06 dist-test-slave-9tkh krb5kdc[10461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:06.286643 10474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:06.287137 10474 webserver.cc:460] Webserver started at http://127.0.0.1:46063/ using document root <none> and password file <none>
I20240501 02:12:06.287322 10474 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:06.287362 10474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:06.287467 10474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:06.288477 10474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "58a1835f46f04e0f891ec2da0b1dcdb0"
format_stamp: "Formatted at 2024-05-01 02:12:06 on dist-test-slave-9tkh"
server_key: "8d0c55ef338f7a3a1d19e3d4dd65aab1"
server_key_iv: "16dfa836c617cbbfd628c459a668e09b"
server_key_version: "encryptionkey@0"
I20240501 02:12:06.288769 10474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "58a1835f46f04e0f891ec2da0b1dcdb0"
format_stamp: "Formatted at 2024-05-01 02:12:06 on dist-test-slave-9tkh"
server_key: "8d0c55ef338f7a3a1d19e3d4dd65aab1"
server_key_iv: "16dfa836c617cbbfd628c459a668e09b"
server_key_version: "encryptionkey@0"
I20240501 02:12:06.289952 10474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:12:06.290592 10489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:06.290760 10474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:06.290843 10474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "58a1835f46f04e0f891ec2da0b1dcdb0"
format_stamp: "Formatted at 2024-05-01 02:12:06 on dist-test-slave-9tkh"
server_key: "8d0c55ef338f7a3a1d19e3d4dd65aab1"
server_key_iv: "16dfa836c617cbbfd628c459a668e09b"
server_key_version: "encryptionkey@0"
I20240501 02:12:06.290899 10474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:06.307672 10474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:06.309556 10474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:06.309659 10474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:06.313011 10474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38889
I20240501 02:12:06.313067 10541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38889 every 8 connection(s)
I20240501 02:12:06.313356 10474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:06.314173 10542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:06.315415 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10474
I20240501 02:12:06.315517 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:06.315665 22577 external_mini_cluster.cc:1428] Setting key a7267fc519a550103733c9fef74f809b
I20240501 02:12:06.316686 10542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0: Bootstrap starting.
I20240501 02:12:06.317340 10542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:06.317600 10542 log.cc:824] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:06.318250 10542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0: No bootstrap required, opened a new log
I20240501 02:12:06.320050 10542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38889 } }
I20240501 02:12:06.320180 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:06.316168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47179 (local address 127.0.0.1:38889)
0501 02:12:06.316362 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:06.316374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:06.316400 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:12:06.316924 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:06.316933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:06.316999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:06.317171 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:06.317486 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:06.318247 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:06.318904 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:06.319158 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:06.319622 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:06.319632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:06.319632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:06.319781 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:06.319803 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:06.319810 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:06.319894 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:06.320006 (+ 112us) server_negotiation.cc:299] Negotiation successful
0501 02:12:06.320032 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20240501 02:12:06.320219 10542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:06.320251 10542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a1835f46f04e0f891ec2da0b1dcdb0, State: Initialized, Role: FOLLOWER
I20240501 02:12:06.320359 10542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38889 } }
I20240501 02:12:06.320426 10542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:06.320462 10542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:06.320509 10542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:06.321105 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:06.321179 10542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38889 } }
I20240501 02:12:06.321285 10542 leader_election.cc:302] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a1835f46f04e0f891ec2da0b1dcdb0; no voters:
I20240501 02:12:06.321440 10542 leader_election.cc:288] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:06.321496 10547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:06.321653 10547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [term 1 LEADER]: Becoming Leader. State: Replica: 58a1835f46f04e0f891ec2da0b1dcdb0, State: Running, Role: LEADER
I20240501 02:12:06.321691 10542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:06.321849 10547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38889 } }
I20240501 02:12:06.322962 10549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38889 } } }
I20240501 02:12:06.323056 10549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:06.323297 10548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a1835f46f04e0f891ec2da0b1dcdb0. Latest consensus state: current_term: 1 leader_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a1835f46f04e0f891ec2da0b1dcdb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38889 } } }
I20240501 02:12:06.323400 10548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0 [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:06.323653 10566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:06.323702 10566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:06.323778 10567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:06.323942 10567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:06.325364 10567 catalog_manager.cc:1347] Generated new cluster ID: f9f66d3b16bf4cac92b42218bc324d7b
I20240501 02:12:06.325416 10567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:06.327090 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:06.323206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54521 (local address 127.0.0.1:38889)
0501 02:12:06.323243 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:06.323245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:06.323635 (+ 390us) server_negotiation.cc:407] Connection header received
0501 02:12:06.323688 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:06.323689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:06.323699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:06.323764 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:06.324933 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:06.325482 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:06.326196 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:06.326354 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:06.326878 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:06.326882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:06.326882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:06.326887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:06.326898 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:06.326902 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:06.326938 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:06.327008 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 02:12:06.327018 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:12:06.335808 10567 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:06.336539 10567 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:06.349925 10567 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 58a1835f46f04e0f891ec2da0b1dcdb0: Generated new TSK 0
I20240501 02:12:06.350147 10567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:06.359212 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10474
W20240501 02:12:06.363481 10459 connection.cc:535] client connection to 127.0.0.1:38889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38889): Connection reset by peer (error 104)
2024-05-01T02:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1179 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:06 dist-test-slave-9tkh krb5kdc[10573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:06 dist-test-slave-9tkh krb5kdc[10573](info): set up 2 sockets
May 01 02:12:06 dist-test-slave-9tkh krb5kdc[10573](info): commencing operation
krb5kdc: starting...
W20240501 02:12:07.254313 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:07 dist-test-slave-9tkh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:07.370265 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:07.448848 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:07.449064 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:07.449100 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:07.450467 10586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:07.450516 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:07.450567 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:07.450587 10586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:07.450613 10586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:07.452289 10586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:07.452527 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:07.452770 10586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:07.456844 10594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:07.457196 10592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:07.457373 10591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:07.457798 10586 server_base.cc:1034] running on GCE node
I20240501 02:12:07.457991 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:07.458215 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:07.459345 10586 hybrid_clock.cc:648] HybridClock initialized: now 1714529527459330 us; error 27 us; skew 500 ppm
May 01 02:12:07 dist-test-slave-9tkh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:07.461612 10586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:07.461997 10586 webserver.cc:460] Webserver started at http://127.0.0.1:45233/ using document root <none> and password file <none>
I20240501 02:12:07.462157 10586 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:07.462196 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:07.462292 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:07.463326 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "d86c9257768a40f98ea397640559f293"
format_stamp: "Formatted at 2024-05-01 02:12:07 on dist-test-slave-9tkh"
server_key: "318b428cdd38f5aa1d5286a9d142a182"
server_key_iv: "6c9cdf5b278b976e621afdc887eaefd0"
server_key_version: "encryptionkey@0"
I20240501 02:12:07.463622 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "d86c9257768a40f98ea397640559f293"
format_stamp: "Formatted at 2024-05-01 02:12:07 on dist-test-slave-9tkh"
server_key: "318b428cdd38f5aa1d5286a9d142a182"
server_key_iv: "6c9cdf5b278b976e621afdc887eaefd0"
server_key_version: "encryptionkey@0"
I20240501 02:12:07.464716 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:12:07.465337 10601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:07.465492 10586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:07.465567 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "d86c9257768a40f98ea397640559f293"
format_stamp: "Formatted at 2024-05-01 02:12:07 on dist-test-slave-9tkh"
server_key: "318b428cdd38f5aa1d5286a9d142a182"
server_key_iv: "6c9cdf5b278b976e621afdc887eaefd0"
server_key_version: "encryptionkey@0"
I20240501 02:12:07.465626 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:07.469090 10586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:07.471117 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:07.471215 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:07.474756 10586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34289
I20240501 02:12:07.474784 10653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34289 every 8 connection(s)
I20240501 02:12:07.475095 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:07.475947 10654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:07.478081 10654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293: Bootstrap starting.
I20240501 02:12:07.478688 10654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:07.478930 10654 log.cc:824] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:07.479581 10654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293: No bootstrap required, opened a new log
I20240501 02:12:07.481019 10654 raft_consensus.cc:348] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c9257768a40f98ea397640559f293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20240501 02:12:07.481154 10654 raft_consensus.cc:374] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:07.481169 10654 raft_consensus.cc:729] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86c9257768a40f98ea397640559f293, State: Initialized, Role: FOLLOWER
I20240501 02:12:07.481292 10654 consensus_queue.cc:260] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c9257768a40f98ea397640559f293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20240501 02:12:07.481360 10654 raft_consensus.cc:388] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:07.481395 10654 raft_consensus.cc:482] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:07.481441 10654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:07.482003 10654 raft_consensus.cc:504] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c9257768a40f98ea397640559f293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20240501 02:12:07.482102 10654 leader_election.cc:302] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d86c9257768a40f98ea397640559f293; no voters:
I20240501 02:12:07.482260 10654 leader_election.cc:288] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:07.482338 10657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:07.482517 10654 sys_catalog.cc:564] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:07.482579 10657 raft_consensus.cc:686] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [term 1 LEADER]: Becoming Leader. State: Replica: d86c9257768a40f98ea397640559f293, State: Running, Role: LEADER
I20240501 02:12:07.482681 10657 consensus_queue.cc:237] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c9257768a40f98ea397640559f293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20240501 02:12:07.483047 10658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d86c9257768a40f98ea397640559f293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c9257768a40f98ea397640559f293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } } }
I20240501 02:12:07.483140 10658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:07.483289 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d86c9257768a40f98ea397640559f293. Latest consensus state: current_term: 1 leader_uuid: "d86c9257768a40f98ea397640559f293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86c9257768a40f98ea397640559f293" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } } }
I20240501 02:12:07.483346 10662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:07.484031 10662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:07.484663 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10586
I20240501 02:12:07.484695 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:07.484771 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:07.484946 22577 external_mini_cluster.cc:1428] Setting key 1ba168a6f712df803778ac83fb688ba8
I20240501 02:12:07.485663 10662 catalog_manager.cc:1347] Generated new cluster ID: 7e04d8f989264645996464ba73a1f3b5
I20240501 02:12:07.485716 10662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:07.489995 10662 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:07.490167 10662 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:07.491763 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:07.485580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57447 (local address 127.0.0.1:34289)
0501 02:12:07.489918 (+ 4338us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:07.489929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:07.489959 (+ 30us) server_negotiation.cc:407] Connection header received
0501 02:12:07.490087 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:07.490094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:07.490128 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:07.490297 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:07.491244 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:07.491247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:07.491248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:07.491253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:07.491283 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:07.491290 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:07.491357 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:07.491431 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 02:12:07.491471 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4294,"thread_start_us":68,"threads_started":1}
I20240501 02:12:07.492691 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:07.495765 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:07.494261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46237 (local address 127.0.0.1:34289)
0501 02:12:07.494299 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:07.494301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:07.494742 (+ 441us) server_negotiation.cc:407] Connection header received
0501 02:12:07.494814 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:07.494815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:07.494820 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:07.494862 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:07.495545 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:07.495546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:07.495547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:07.495551 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:07.495567 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:07.495571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:07.495617 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:07.495675 (+ 58us) server_negotiation.cc:299] Negotiation successful
0501 02:12:07.495689 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:07.501013 10662 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d86c9257768a40f98ea397640559f293: Generated new TSK 0
I20240501 02:12:07.501212 10662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:07.502839 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10586
2024-05-01T02:12:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:07 dist-test-slave-9tkh krb5kdc[10685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:07 dist-test-slave-9tkh krb5kdc[10685](info): set up 2 sockets
May 01 02:12:07 dist-test-slave-9tkh krb5kdc[10685](info): commencing operation
krb5kdc: starting...
W20240501 02:12:08.398293 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:08 dist-test-slave-9tkh krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:08.509701 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:08.587924 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:08.588137 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:08.588160 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:08.589495 10698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:08.589541 10698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:08.589555 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:08.589565 10698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:08.589581 10698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:08.591334 10698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:08.591533 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:08.591694 10698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:08.595575 10704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:08.595885 10703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:08.596285 10706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:08.596683 10698 server_base.cc:1034] running on GCE node
I20240501 02:12:08.596909 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:08.597139 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:08.598280 10698 hybrid_clock.cc:648] HybridClock initialized: now 1714529528598259 us; error 34 us; skew 500 ppm
May 01 02:12:08 dist-test-slave-9tkh krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:08.600630 10698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:08.601054 10698 webserver.cc:460] Webserver started at http://127.0.0.1:40207/ using document root <none> and password file <none>
I20240501 02:12:08.601219 10698 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:08.601258 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:08.601362 10698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:08.602332 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b"
format_stamp: "Formatted at 2024-05-01 02:12:08 on dist-test-slave-9tkh"
server_key: "ec377db0b4afd01688324f277be41a88"
server_key_iv: "bfc1079c5d43ef371efb43e845a388da"
server_key_version: "encryptionkey@0"
I20240501 02:12:08.602650 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b"
format_stamp: "Formatted at 2024-05-01 02:12:08 on dist-test-slave-9tkh"
server_key: "ec377db0b4afd01688324f277be41a88"
server_key_iv: "bfc1079c5d43ef371efb43e845a388da"
server_key_version: "encryptionkey@0"
I20240501 02:12:08.603786 10698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:08.604410 10713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:08.604555 10698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:08.604626 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b"
format_stamp: "Formatted at 2024-05-01 02:12:08 on dist-test-slave-9tkh"
server_key: "ec377db0b4afd01688324f277be41a88"
server_key_iv: "bfc1079c5d43ef371efb43e845a388da"
server_key_version: "encryptionkey@0"
I20240501 02:12:08.604702 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:08.620296 10698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:08.621975 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:08.622085 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:08.625851 10698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40417
I20240501 02:12:08.625885 10765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40417 every 8 connection(s)
I20240501 02:12:08.626201 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:08.627017 10766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:08.629068 10766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b: Bootstrap starting.
I20240501 02:12:08.629571 10766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:08.629786 10766 log.cc:824] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:08.630405 10766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b: No bootstrap required, opened a new log
I20240501 02:12:08.631842 10766 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40417 } }
I20240501 02:12:08.631970 10766 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:08.631986 10766 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec7dd1ba1ac44dfb88be637b44e4f87b, State: Initialized, Role: FOLLOWER
I20240501 02:12:08.632100 10766 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40417 } }
I20240501 02:12:08.632169 10766 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:08.632197 10766 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:08.632228 10766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:08.632715 10766 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40417 } }
I20240501 02:12:08.632799 10766 leader_election.cc:302] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec7dd1ba1ac44dfb88be637b44e4f87b; no voters:
I20240501 02:12:08.632936 10766 leader_election.cc:288] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:08.632992 10769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:08.633153 10769 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [term 1 LEADER]: Becoming Leader. State: Replica: ec7dd1ba1ac44dfb88be637b44e4f87b, State: Running, Role: LEADER
I20240501 02:12:08.633234 10769 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40417 } }
I20240501 02:12:08.633483 10766 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:08.633657 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40417 } } }
I20240501 02:12:08.633733 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:08.633786 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10698
I20240501 02:12:08.633906 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:08.634095 22577 external_mini_cluster.cc:1428] Setting key c61d579a9e85fa3ca218650d51ce30a2
I20240501 02:12:08.634289 10773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:08.634186 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7dd1ba1ac44dfb88be637b44e4f87b. Latest consensus state: current_term: 1 leader_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7dd1ba1ac44dfb88be637b44e4f87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40417 } } }
I20240501 02:12:08.634675 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:08.634928 10773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:08.639233 10773 catalog_manager.cc:1347] Generated new cluster ID: 5e59959288344525b9937a2cfe07c590
I20240501 02:12:08.639297 10773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:12:08 dist-test-slave-9tkh krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 02:12:08.644371 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:08.634551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37303 (local address 127.0.0.1:40417)
0501 02:12:08.635066 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:08.635079 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:08.635102 (+ 23us) server_negotiation.cc:407] Connection header received
0501 02:12:08.635224 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:08.635231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:08.635282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:08.635412 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:08.636315 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:08.637066 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:08.637819 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:08.638016 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:08.640422 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:08.640427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:08.640428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:08.640449 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:08.642954 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:08.643338 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:08.643341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:08.643341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:08.643398 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:08.643553 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:08.643554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:08.643554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:08.643781 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:08.643905 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:08.644038 (+ 133us) server_negotiation.cc:299] Negotiation successful
0501 02:12:08.644162 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":75,"threads_started":1}
I20240501 02:12:08.645370 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:08.657371 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:08.650890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47881 (local address 127.0.0.1:40417)
0501 02:12:08.650937 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:08.650939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:08.651025 (+ 86us) server_negotiation.cc:407] Connection header received
0501 02:12:08.651078 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:08.651079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:08.651090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:08.651162 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:08.651946 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:08.652687 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:08.653361 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:08.653947 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:08.654667 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:08.654670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:08.654671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:08.654676 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:08.656387 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:08.656726 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:08.656727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:08.656727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:08.656751 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:08.656878 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:08.656878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:08.656879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:08.656989 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:08.657076 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:08.657179 (+ 103us) server_negotiation.cc:299] Negotiation successful
0501 02:12:08.657265 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 02:12:08.666205 10773 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:08.667066 10773 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:08.680506 10773 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ec7dd1ba1ac44dfb88be637b44e4f87b: Generated new TSK 0
I20240501 02:12:08.680773 10773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:08.689513 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10698
2024-05-01T02:12:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:08 dist-test-slave-9tkh krb5kdc[10797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:08 dist-test-slave-9tkh krb5kdc[10797](info): set up 2 sockets
May 01 02:12:08 dist-test-slave-9tkh krb5kdc[10797](info): commencing operation
krb5kdc: starting...
W20240501 02:12:09.586359 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:09 dist-test-slave-9tkh krb5kdc[10797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:09.700875 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:09.781572 10810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:09.781806 10810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:09.781842 10810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:09.783250 10810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:09.783305 10810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:09.783324 10810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:09.783339 10810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:09.783351 10810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:09.785113 10810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:09.785354 10810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:09.785549 10810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:09.789932 10818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:09.790268 10810 server_base.cc:1034] running on GCE node
W20240501 02:12:09.790385 10816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:09.790746 10815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:09.791049 10810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:09.791296 10810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:09.792420 10810 hybrid_clock.cc:648] HybridClock initialized: now 1714529529792406 us; error 39 us; skew 500 ppm
May 01 02:12:09 dist-test-slave-9tkh krb5kdc[10797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:09.794914 10810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:09.795325 10810 webserver.cc:460] Webserver started at http://127.0.0.1:45027/ using document root <none> and password file <none>
I20240501 02:12:09.795493 10810 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:09.795533 10810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:09.795617 10810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:09.796617 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "cc46f1b285b64bba809cc18b12077836"
format_stamp: "Formatted at 2024-05-01 02:12:09 on dist-test-slave-9tkh"
server_key: "9fa52c34898356a9a52d5367cc8325e5"
server_key_iv: "0225a906a207300286b460ce7771a54f"
server_key_version: "encryptionkey@0"
I20240501 02:12:09.796916 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "cc46f1b285b64bba809cc18b12077836"
format_stamp: "Formatted at 2024-05-01 02:12:09 on dist-test-slave-9tkh"
server_key: "9fa52c34898356a9a52d5367cc8325e5"
server_key_iv: "0225a906a207300286b460ce7771a54f"
server_key_version: "encryptionkey@0"
I20240501 02:12:09.798031 10810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:09.798651 10825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:09.798815 10810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:09.798897 10810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "cc46f1b285b64bba809cc18b12077836"
format_stamp: "Formatted at 2024-05-01 02:12:09 on dist-test-slave-9tkh"
server_key: "9fa52c34898356a9a52d5367cc8325e5"
server_key_iv: "0225a906a207300286b460ce7771a54f"
server_key_version: "encryptionkey@0"
I20240501 02:12:09.798956 10810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:09.814893 10810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:09.816794 10810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:09.816916 10810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:09.820585 10810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38883
I20240501 02:12:09.820626 10877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38883 every 8 connection(s)
I20240501 02:12:09.820935 10810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:09.821563 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:09.823801 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836: Bootstrap starting.
I20240501 02:12:09.824352 10878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:09.824596 10878 log.cc:824] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:09.825248 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836: No bootstrap required, opened a new log
I20240501 02:12:09.825452 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10810
I20240501 02:12:09.825551 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:09.825691 22577 external_mini_cluster.cc:1428] Setting key b58f061ea3a97c838f07794de6a90fcf
I20240501 02:12:09.827145 10878 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46f1b285b64bba809cc18b12077836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38883 } }
I20240501 02:12:09.827301 10878 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:09.827328 10878 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc46f1b285b64bba809cc18b12077836, State: Initialized, Role: FOLLOWER
I20240501 02:12:09.827473 10878 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46f1b285b64bba809cc18b12077836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38883 } }
I20240501 02:12:09.827546 10878 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:09.827567 10878 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:09.827610 10878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:09.828140 10878 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46f1b285b64bba809cc18b12077836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38883 } }
I20240501 02:12:09.828224 10878 leader_election.cc:302] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc46f1b285b64bba809cc18b12077836; no voters:
I20240501 02:12:09.828375 10878 leader_election.cc:288] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:09.828435 10883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:09.828555 10883 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [term 1 LEADER]: Becoming Leader. State: Replica: cc46f1b285b64bba809cc18b12077836, State: Running, Role: LEADER
I20240501 02:12:09.828619 10878 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:09.828701 10883 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46f1b285b64bba809cc18b12077836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38883 } }
I20240501 02:12:09.829020 10883 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc46f1b285b64bba809cc18b12077836. Latest consensus state: current_term: 1 leader_uuid: "cc46f1b285b64bba809cc18b12077836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46f1b285b64bba809cc18b12077836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38883 } } }
I20240501 02:12:09.829123 10883 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:09.829030 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc46f1b285b64bba809cc18b12077836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46f1b285b64bba809cc18b12077836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38883 } } }
I20240501 02:12:09.829250 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836 [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:09.830370 10899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:09.830425 10899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:09.830613 10890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:09.830801 10890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:09.831380 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:09.826165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34679 (local address 127.0.0.1:38883)
0501 02:12:09.826342 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:09.826354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:09.826379 (+ 25us) server_negotiation.cc:407] Connection header received
0501 02:12:09.827117 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:09.827126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:09.827191 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:09.827353 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:09.827731 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:09.829087 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:09.829871 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:09.830095 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:09.830761 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:09.830769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:09.830769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:09.830943 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:09.830965 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:09.830973 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:09.831042 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:09.831149 (+ 107us) server_negotiation.cc:299] Negotiation successful
0501 02:12:09.831198 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20240501 02:12:09.832243 10890 catalog_manager.cc:1347] Generated new cluster ID: 26086aaaac9642ddaf38f2d355aeb817
I20240501 02:12:09.832300 10890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:09.834682 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:09.840958 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:09.836055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41291 (local address 127.0.0.1:38883)
0501 02:12:09.836098 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:09.836101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:09.838737 (+ 2636us) server_negotiation.cc:407] Connection header received
0501 02:12:09.838824 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:09.838826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:09.838839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:09.838885 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:09.839150 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:09.839639 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:09.840143 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:09.840609 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:09.840728 (+ 119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:09.840732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:09.840732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:09.840736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:09.840749 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:09.840752 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:09.840792 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:09.840849 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 02:12:09.840876 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:12:09.858651 10890 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:09.859206 10890 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:09.872406 10890 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cc46f1b285b64bba809cc18b12077836: Generated new TSK 0
I20240501 02:12:09.872622 10890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:09.876178 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10810
2024-05-01T02:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:09 dist-test-slave-9tkh krb5kdc[10909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:09 dist-test-slave-9tkh krb5kdc[10909](info): set up 2 sockets
May 01 02:12:09 dist-test-slave-9tkh krb5kdc[10909](info): commencing operation
krb5kdc: starting...
W20240501 02:12:10.774396 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:10 dist-test-slave-9tkh krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:10.890399 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:10.969911 10922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:10.970113 10922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:10.970144 10922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:10.971520 10922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:10.971571 10922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:10.971589 10922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:10.971604 10922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:10.971618 10922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:10.973289 10922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:10.973526 10922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:10.973704 10922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:10.977720 10927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:10.978101 10930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:10.978580 10928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:10.979068 10922 server_base.cc:1034] running on GCE node
I20240501 02:12:10.979315 10922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:10.979558 10922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:10.980712 10922 hybrid_clock.cc:648] HybridClock initialized: now 1714529530980701 us; error 27 us; skew 500 ppm
May 01 02:12:10 dist-test-slave-9tkh krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:10.983223 10922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:10.983642 10922 webserver.cc:460] Webserver started at http://127.0.0.1:32827/ using document root <none> and password file <none>
I20240501 02:12:10.983829 10922 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:10.983871 10922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:10.983978 10922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:10.985025 10922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "e80b5427db634841b21cd309d79eb05e"
format_stamp: "Formatted at 2024-05-01 02:12:10 on dist-test-slave-9tkh"
server_key: "a9a5f40f374e2fbef96e8d132a4ea9a4"
server_key_iv: "fcb4a8d3109d7624674e05db384a2b81"
server_key_version: "encryptionkey@0"
I20240501 02:12:10.985311 10922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "e80b5427db634841b21cd309d79eb05e"
format_stamp: "Formatted at 2024-05-01 02:12:10 on dist-test-slave-9tkh"
server_key: "a9a5f40f374e2fbef96e8d132a4ea9a4"
server_key_iv: "fcb4a8d3109d7624674e05db384a2b81"
server_key_version: "encryptionkey@0"
I20240501 02:12:10.986409 10922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:12:10.987152 10937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:10.987313 10922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:10.987391 10922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "e80b5427db634841b21cd309d79eb05e"
format_stamp: "Formatted at 2024-05-01 02:12:10 on dist-test-slave-9tkh"
server_key: "a9a5f40f374e2fbef96e8d132a4ea9a4"
server_key_iv: "fcb4a8d3109d7624674e05db384a2b81"
server_key_version: "encryptionkey@0"
I20240501 02:12:10.987450 10922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:10.991293 10922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:10.993508 10922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:10.993626 10922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:10.997165 10922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32963
I20240501 02:12:10.997195 10989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32963 every 8 connection(s)
I20240501 02:12:10.997488 10922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:10.998310 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:11.000372 10990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e: Bootstrap starting.
I20240501 02:12:11.000864 10990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:11.001072 10990 log.cc:824] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:11.001645 10990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e: No bootstrap required, opened a new log
I20240501 02:12:11.003080 10990 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b5427db634841b21cd309d79eb05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32963 } }
I20240501 02:12:11.003198 10990 raft_consensus.cc:374] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:11.003214 10990 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80b5427db634841b21cd309d79eb05e, State: Initialized, Role: FOLLOWER
I20240501 02:12:11.003311 10990 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b5427db634841b21cd309d79eb05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32963 } }
I20240501 02:12:11.003376 10990 raft_consensus.cc:388] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:11.003403 10990 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:11.003432 10990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:11.003917 10990 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b5427db634841b21cd309d79eb05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32963 } }
I20240501 02:12:11.003995 10990 leader_election.cc:302] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e80b5427db634841b21cd309d79eb05e; no voters:
I20240501 02:12:11.004118 10990 leader_election.cc:288] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:11.004191 10993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:11.004344 10990 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:11.004349 10993 raft_consensus.cc:686] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [term 1 LEADER]: Becoming Leader. State: Replica: e80b5427db634841b21cd309d79eb05e, State: Running, Role: LEADER
I20240501 02:12:11.004504 10993 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b5427db634841b21cd309d79eb05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32963 } }
I20240501 02:12:11.004967 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 10922
I20240501 02:12:11.004987 10995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80b5427db634841b21cd309d79eb05e. Latest consensus state: current_term: 1 leader_uuid: "e80b5427db634841b21cd309d79eb05e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b5427db634841b21cd309d79eb05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32963 } } }
I20240501 02:12:11.005077 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:11.005084 10995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:11.005000 10994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e80b5427db634841b21cd309d79eb05e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b5427db634841b21cd309d79eb05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32963 } } }
I20240501 02:12:11.005136 10994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:11.005252 22577 external_mini_cluster.cc:1428] Setting key 838fde251d640594d344a7390064838e
I20240501 02:12:11.005268 10998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:11.005765 10998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:11.007517 10998 catalog_manager.cc:1347] Generated new cluster ID: 86ba3852f85f4f23938da958271a21d6
I20240501 02:12:11.007573 10998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:11.007835 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:11.005876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58489 (local address 127.0.0.1:32963)
0501 02:12:11.006344 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:11.006354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:11.006370 (+ 16us) server_negotiation.cc:407] Connection header received
0501 02:12:11.006463 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:11.006469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:11.006499 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:11.006666 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:11.007393 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:11.007396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:11.007396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:11.007400 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:11.007423 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:11.007430 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:11.007500 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:11.007668 (+ 168us) server_negotiation.cc:299] Negotiation successful
0501 02:12:11.007704 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":75,"threads_started":1}
I20240501 02:12:11.008566 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:11.010742 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:11.009746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37285 (local address 127.0.0.1:32963)
0501 02:12:11.009778 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:11.009779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:11.009892 (+ 113us) server_negotiation.cc:407] Connection header received
0501 02:12:11.009931 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:11.009932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:11.009936 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:11.009986 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:11.010507 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:11.010508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:11.010508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:11.010512 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:11.010524 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:11.010548 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:11.010592 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:11.010671 (+ 79us) server_negotiation.cc:299] Negotiation successful
0501 02:12:11.010684 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:12:11.017200 10998 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:11.017346 10998 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:11.024813 10998 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e80b5427db634841b21cd309d79eb05e: Generated new TSK 0
I20240501 02:12:11.024998 10998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:11.029945 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 10922
2024-05-01T02:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:11 dist-test-slave-9tkh krb5kdc[11021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:11 dist-test-slave-9tkh krb5kdc[11021](info): set up 2 sockets
May 01 02:12:11 dist-test-slave-9tkh krb5kdc[11021](info): commencing operation
krb5kdc: starting...
W20240501 02:12:11.918375 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:11 dist-test-slave-9tkh krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:12.030992 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:12.110522 11034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:12.110764 11034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:12.110795 11034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:12.112152 11034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:12.112200 11034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:12.112212 11034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:12.112223 11034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:12.112241 11034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:12.113925 11034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:12.114141 11034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:12.114321 11034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:12.118186 11039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:12.118592 11040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:12.118922 11042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:12.119324 11034 server_base.cc:1034] running on GCE node
I20240501 02:12:12.119525 11034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:12.119761 11034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:12.120899 11034 hybrid_clock.cc:648] HybridClock initialized: now 1714529532120881 us; error 34 us; skew 500 ppm
May 01 02:12:12 dist-test-slave-9tkh krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:12.128831 11034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:12.129259 11034 webserver.cc:460] Webserver started at http://127.0.0.1:40167/ using document root <none> and password file <none>
I20240501 02:12:12.129426 11034 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:12.129460 11034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:12.129547 11034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:12.130585 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "9cb64e54cbb34548bc1986c80e630840"
format_stamp: "Formatted at 2024-05-01 02:12:12 on dist-test-slave-9tkh"
server_key: "1c687552c115b6d85aef43fae7e7676d"
server_key_iv: "ca7101d9242965e8d9cbc2118a005a24"
server_key_version: "encryptionkey@0"
I20240501 02:12:12.130879 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "9cb64e54cbb34548bc1986c80e630840"
format_stamp: "Formatted at 2024-05-01 02:12:12 on dist-test-slave-9tkh"
server_key: "1c687552c115b6d85aef43fae7e7676d"
server_key_iv: "ca7101d9242965e8d9cbc2118a005a24"
server_key_version: "encryptionkey@0"
I20240501 02:12:12.132004 11034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:12.132618 11049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:12.132784 11034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:12.132879 11034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "9cb64e54cbb34548bc1986c80e630840"
format_stamp: "Formatted at 2024-05-01 02:12:12 on dist-test-slave-9tkh"
server_key: "1c687552c115b6d85aef43fae7e7676d"
server_key_iv: "ca7101d9242965e8d9cbc2118a005a24"
server_key_version: "encryptionkey@0"
I20240501 02:12:12.132941 11034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:12.150790 11034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:12.152886 11034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:12.152988 11034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:12.156625 11034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43549
I20240501 02:12:12.156692 11101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43549 every 8 connection(s)
I20240501 02:12:12.156981 11034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:12.157728 11102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:12.159905 11102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840: Bootstrap starting.
I20240501 02:12:12.160445 11102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:12.160687 11102 log.cc:824] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:12.161343 11102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840: No bootstrap required, opened a new log
I20240501 02:12:12.162930 11102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb64e54cbb34548bc1986c80e630840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20240501 02:12:12.163057 11102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:12.163074 11102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb64e54cbb34548bc1986c80e630840, State: Initialized, Role: FOLLOWER
I20240501 02:12:12.163192 11102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb64e54cbb34548bc1986c80e630840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20240501 02:12:12.163250 11102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:12.163269 11102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:12.163295 11102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:12.163834 11102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb64e54cbb34548bc1986c80e630840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20240501 02:12:12.163920 11102 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb64e54cbb34548bc1986c80e630840; no voters:
I20240501 02:12:12.164090 11102 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:12.164165 11105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:12.164355 11105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [term 1 LEADER]: Becoming Leader. State: Replica: 9cb64e54cbb34548bc1986c80e630840, State: Running, Role: LEADER
I20240501 02:12:12.164476 11105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb64e54cbb34548bc1986c80e630840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20240501 02:12:12.164659 11102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:12.164835 11105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb64e54cbb34548bc1986c80e630840. Latest consensus state: current_term: 1 leader_uuid: "9cb64e54cbb34548bc1986c80e630840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb64e54cbb34548bc1986c80e630840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20240501 02:12:12.164921 11105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:12.164853 11106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cb64e54cbb34548bc1986c80e630840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb64e54cbb34548bc1986c80e630840" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20240501 02:12:12.165067 11106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:12.165199 11109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:12.165884 11109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:12.165980 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11034
I20240501 02:12:12.166093 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:12.166265 22577 external_mini_cluster.cc:1428] Setting key 36425f78eb3f9cf270c569d0cdcd4d47
I20240501 02:12:12.167551 11109 catalog_manager.cc:1347] Generated new cluster ID: 1874481b88c649489118b81c77dfba9e
I20240501 02:12:12.167599 11109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:12:12 dist-test-slave-9tkh krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 02:12:12.177778 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:12.166914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:45801 (local address 127.0.0.1:43549)
0501 02:12:12.167054 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:12.167066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:12.167217 (+ 151us) server_negotiation.cc:407] Connection header received
0501 02:12:12.167328 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:12.167334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:12.167387 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:12.168488 (+ 1101us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:12.168633 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:12.169759 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:12.170580 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:12.170847 (+ 267us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:12.173370 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:12.173378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:12.173379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:12.173405 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:12.176150 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:12.176641 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:12.176644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:12.176645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:12.176707 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:12.176908 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:12.176909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:12.176909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:12.177153 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:12.177297 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:12.177450 (+ 153us) server_negotiation.cc:299] Negotiation successful
0501 02:12:12.177560 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20240501 02:12:12.178966 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:12.187754 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:12.180470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:40189 (local address 127.0.0.1:43549)
0501 02:12:12.180509 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:12.180511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:12.180667 (+ 156us) server_negotiation.cc:407] Connection header received
0501 02:12:12.180734 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:12.180734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:12.180744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:12.180808 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:12.181689 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:12.182408 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:12.183196 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:12.183374 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:12.184942 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:12.184947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:12.184947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:12.184953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:12.186609 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:12.187040 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:12.187041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:12.187041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:12.187063 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:12.187235 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:12.187236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:12.187236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:12.187350 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:12.187458 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:12.187581 (+ 123us) server_negotiation.cc:299] Negotiation successful
0501 02:12:12.187647 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:12:12.202934 11109 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:12.203523 11109 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:12.209388 11109 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9cb64e54cbb34548bc1986c80e630840: Generated new TSK 0
I20240501 02:12:12.209581 11109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:12.214133 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11034
W20240501 02:12:12.218235 11019 connection.cc:535] client connection to 127.0.0.1:43549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43549): Connection reset by peer (error 104)
2024-05-01T02:12:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:12 dist-test-slave-9tkh krb5kdc[11133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:12 dist-test-slave-9tkh krb5kdc[11133](info): set up 2 sockets
May 01 02:12:12 dist-test-slave-9tkh krb5kdc[11133](info): commencing operation
krb5kdc: starting...
W20240501 02:12:13.102463 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:13 dist-test-slave-9tkh krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:13.221350 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:13.300380 11146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:13.300592 11146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:13.300613 11146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:13.301947 11146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:13.301998 11146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:13.302011 11146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:13.302021 11146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:13.302031 11146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:13.303704 11146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:13.303977 11146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:13.304201 11146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:13.308506 11151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:13.308846 11152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:13.309280 11154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:13.309610 11146 server_base.cc:1034] running on GCE node
I20240501 02:12:13.309854 11146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:13.310097 11146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:13.311226 11146 hybrid_clock.cc:648] HybridClock initialized: now 1714529533311207 us; error 33 us; skew 500 ppm
May 01 02:12:13 dist-test-slave-9tkh krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:13.313566 11146 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:13.313968 11146 webserver.cc:460] Webserver started at http://127.0.0.1:38949/ using document root <none> and password file <none>
I20240501 02:12:13.314132 11146 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:13.314167 11146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:13.314251 11146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:13.315265 11146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "513364a740964b39b0f595c0cdc36d1f"
format_stamp: "Formatted at 2024-05-01 02:12:13 on dist-test-slave-9tkh"
server_key: "b5e0056fd80171a6cdf4069764a8dacd"
server_key_iv: "9ce2325ca3a7d93ce097357d9a4df3a2"
server_key_version: "encryptionkey@0"
I20240501 02:12:13.315531 11146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "513364a740964b39b0f595c0cdc36d1f"
format_stamp: "Formatted at 2024-05-01 02:12:13 on dist-test-slave-9tkh"
server_key: "b5e0056fd80171a6cdf4069764a8dacd"
server_key_iv: "9ce2325ca3a7d93ce097357d9a4df3a2"
server_key_version: "encryptionkey@0"
I20240501 02:12:13.316648 11146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:12:13.317312 11161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:13.317483 11146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:13.317560 11146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "513364a740964b39b0f595c0cdc36d1f"
format_stamp: "Formatted at 2024-05-01 02:12:13 on dist-test-slave-9tkh"
server_key: "b5e0056fd80171a6cdf4069764a8dacd"
server_key_iv: "9ce2325ca3a7d93ce097357d9a4df3a2"
server_key_version: "encryptionkey@0"
I20240501 02:12:13.317618 11146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:13.336993 11146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:13.339013 11146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:13.339121 11146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:13.342763 11146 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44959
I20240501 02:12:13.342808 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44959 every 8 connection(s)
I20240501 02:12:13.343111 11146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:13.343808 11214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:13.345911 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11146
I20240501 02:12:13.345957 11214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f: Bootstrap starting.
I20240501 02:12:13.346004 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:13.346148 22577 external_mini_cluster.cc:1428] Setting key 9fca2f45f22b5b8ce7de2cbd4e82f0e7
I20240501 02:12:13.346623 11214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:13.346936 11214 log.cc:824] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:13.347628 11214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f: No bootstrap required, opened a new log
I20240501 02:12:13.349627 11214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513364a740964b39b0f595c0cdc36d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20240501 02:12:13.349771 11214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:13.349798 11214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513364a740964b39b0f595c0cdc36d1f, State: Initialized, Role: FOLLOWER
I20240501 02:12:13.349898 11214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513364a740964b39b0f595c0cdc36d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20240501 02:12:13.349972 11214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:13.349999 11214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:13.350037 11214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:13.350680 11214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513364a740964b39b0f595c0cdc36d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20240501 02:12:13.350793 11214 leader_election.cc:302] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 513364a740964b39b0f595c0cdc36d1f; no voters:
I20240501 02:12:13.350966 11214 leader_election.cc:288] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:13.351037 11219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:13.351245 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:13.346674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:43447 (local address 127.0.0.1:44959)
0501 02:12:13.346859 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:13.346875 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:13.346901 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:12:13.347427 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:13.347437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:13.347498 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:13.347655 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:13.347991 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:13.349173 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:13.349758 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:13.349994 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:13.350438 (+ 444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:13.350447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:13.350448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:13.350647 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:13.350677 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:13.350685 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:13.350806 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:13.350950 (+ 144us) server_negotiation.cc:299] Negotiation successful
0501 02:12:13.350976 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20240501 02:12:13.351217 11214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:13.351217 11219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [term 1 LEADER]: Becoming Leader. State: Replica: 513364a740964b39b0f595c0cdc36d1f, State: Running, Role: LEADER
I20240501 02:12:13.351550 11219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513364a740964b39b0f595c0cdc36d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } }
I20240501 02:12:13.351902 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 513364a740964b39b0f595c0cdc36d1f. Latest consensus state: current_term: 1 leader_uuid: "513364a740964b39b0f595c0cdc36d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513364a740964b39b0f595c0cdc36d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } } }
I20240501 02:12:13.351986 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:13.352190 11228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:13.352373 11220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "513364a740964b39b0f595c0cdc36d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513364a740964b39b0f595c0cdc36d1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44959 } } }
I20240501 02:12:13.352439 11220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:13.352807 11228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:13.353621 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:13.354482 11228 catalog_manager.cc:1347] Generated new cluster ID: 38ec6df222474482a753b7f6137cf2eb
I20240501 02:12:13.354560 11228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:13.357466 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:13.355184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:49935 (local address 127.0.0.1:44959)
0501 02:12:13.355219 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:13.355221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:13.355294 (+ 73us) server_negotiation.cc:407] Connection header received
0501 02:12:13.355360 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:13.355361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:13.355371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:13.355421 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:13.355685 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:13.356126 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:13.356617 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:13.356769 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:13.357229 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:13.357232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:13.357232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:13.357237 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:13.357258 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:13.357262 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:13.357314 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:13.357380 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 02:12:13.357390 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:12:13.366876 11228 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:13.367457 11228 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:13.373869 11228 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 513364a740964b39b0f595c0cdc36d1f: Generated new TSK 0
I20240501 02:12:13.374079 11228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:13.381132 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11146
W20240501 02:12:13.385339 11131 connection.cc:535] client connection to 127.0.0.1:44959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44959): Connection reset by peer (error 104)
2024-05-01T02:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1167 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:13 dist-test-slave-9tkh krb5kdc[11245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:13 dist-test-slave-9tkh krb5kdc[11245](info): set up 2 sockets
May 01 02:12:13 dist-test-slave-9tkh krb5kdc[11245](info): commencing operation
krb5kdc: starting...
W20240501 02:12:14.286470 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.888s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:14 dist-test-slave-9tkh krb5kdc[11245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:14.397428 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:14.476029 11258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:14.476246 11258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:14.476270 11258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:14.477635 11258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:14.477687 11258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:14.477700 11258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:14.477711 11258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:14.477721 11258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:14.479470 11258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:14.479676 11258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:14.479862 11258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:14.483783 11264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:14.484084 11266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:14.484583 11263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:14.485031 11258 server_base.cc:1034] running on GCE node
I20240501 02:12:14.485275 11258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:14.485522 11258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:14.486668 11258 hybrid_clock.cc:648] HybridClock initialized: now 1714529534486656 us; error 31 us; skew 500 ppm
May 01 02:12:14 dist-test-slave-9tkh krb5kdc[11245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:14.489027 11258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:14.489459 11258 webserver.cc:460] Webserver started at http://127.0.0.1:35781/ using document root <none> and password file <none>
I20240501 02:12:14.489627 11258 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:14.489670 11258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:14.489777 11258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:14.490818 11258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "31d35874bd0545fdac08189745936a6c"
format_stamp: "Formatted at 2024-05-01 02:12:14 on dist-test-slave-9tkh"
server_key: "337b2b7350dd390a94cabcb4caac5997"
server_key_iv: "ed9ac6d2a27501375375f2f522025cb2"
server_key_version: "encryptionkey@0"
I20240501 02:12:14.491108 11258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "31d35874bd0545fdac08189745936a6c"
format_stamp: "Formatted at 2024-05-01 02:12:14 on dist-test-slave-9tkh"
server_key: "337b2b7350dd390a94cabcb4caac5997"
server_key_iv: "ed9ac6d2a27501375375f2f522025cb2"
server_key_version: "encryptionkey@0"
I20240501 02:12:14.492316 11258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:14.493063 11273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:14.493240 11258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:14.493319 11258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "31d35874bd0545fdac08189745936a6c"
format_stamp: "Formatted at 2024-05-01 02:12:14 on dist-test-slave-9tkh"
server_key: "337b2b7350dd390a94cabcb4caac5997"
server_key_iv: "ed9ac6d2a27501375375f2f522025cb2"
server_key_version: "encryptionkey@0"
I20240501 02:12:14.493377 11258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:14.496860 11258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:14.498589 11258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:14.498701 11258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:14.502439 11258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38285
I20240501 02:12:14.502497 11325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38285 every 8 connection(s)
I20240501 02:12:14.502863 11258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:14.503777 11326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:14.505916 11326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c: Bootstrap starting.
I20240501 02:12:14.506458 11326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:14.506738 11326 log.cc:824] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:14.507372 11326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c: No bootstrap required, opened a new log
I20240501 02:12:14.508720 11326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d35874bd0545fdac08189745936a6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38285 } }
I20240501 02:12:14.508841 11326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:14.508857 11326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d35874bd0545fdac08189745936a6c, State: Initialized, Role: FOLLOWER
I20240501 02:12:14.508968 11326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d35874bd0545fdac08189745936a6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38285 } }
I20240501 02:12:14.509042 11326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:14.509068 11326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:14.509097 11326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:14.509593 11326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d35874bd0545fdac08189745936a6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38285 } }
I20240501 02:12:14.509677 11326 leader_election.cc:302] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d35874bd0545fdac08189745936a6c; no voters:
I20240501 02:12:14.509795 11326 leader_election.cc:288] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:14.509851 11329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:14.510010 11329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [term 1 LEADER]: Becoming Leader. State: Replica: 31d35874bd0545fdac08189745936a6c, State: Running, Role: LEADER
I20240501 02:12:14.510093 11329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d35874bd0545fdac08189745936a6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38285 } }
I20240501 02:12:14.510110 11326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:14.510459 11330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d35874bd0545fdac08189745936a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d35874bd0545fdac08189745936a6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38285 } } }
I20240501 02:12:14.510569 11330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:14.510668 11331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d35874bd0545fdac08189745936a6c. Latest consensus state: current_term: 1 leader_uuid: "31d35874bd0545fdac08189745936a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d35874bd0545fdac08189745936a6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38285 } } }
I20240501 02:12:14.510728 11331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:14.511034 11334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:14.511471 11334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:14.511780 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11258
I20240501 02:12:14.511898 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:14.512054 22577 external_mini_cluster.cc:1428] Setting key 195101597af71320bee0969ee08673bd
I20240501 02:12:14.513504 11334 catalog_manager.cc:1347] Generated new cluster ID: 389d0ef75cde410a81c28e446598f141
I20240501 02:12:14.513551 11334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:14.520674 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:14.512552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:38123 (local address 127.0.0.1:38285)
0501 02:12:14.518895 (+ 6343us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:14.518907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:14.518930 (+ 23us) server_negotiation.cc:407] Connection header received
0501 02:12:14.519277 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:14.519283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:14.519313 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:14.519459 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:14.520172 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:14.520175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:14.520176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:14.520180 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:14.520204 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:14.520211 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:14.520283 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:14.520453 (+ 170us) server_negotiation.cc:299] Negotiation successful
0501 02:12:14.520492 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6294,"thread_start_us":78,"threads_started":1}
I20240501 02:12:14.521534 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:14.522711 11334 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:14.522855 11334 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:14.524132 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:14.523067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:59661 (local address 127.0.0.1:38285)
0501 02:12:14.523100 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:14.523102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:14.523210 (+ 108us) server_negotiation.cc:407] Connection header received
0501 02:12:14.523270 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:14.523271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:14.523276 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:14.523331 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:14.523935 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:14.523936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:14.523936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:14.523940 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:14.523955 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:14.523958 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:14.524005 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:14.524058 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 02:12:14.524070 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:12:14.527045 11334 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 31d35874bd0545fdac08189745936a6c: Generated new TSK 0
I20240501 02:12:14.527209 11334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:14.531313 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11258
2024-05-01T02:12:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1150 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:14 dist-test-slave-9tkh krb5kdc[11357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:14 dist-test-slave-9tkh krb5kdc[11357](info): set up 2 sockets
May 01 02:12:14 dist-test-slave-9tkh krb5kdc[11357](info): commencing operation
krb5kdc: starting...
W20240501 02:12:15.418464 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:15 dist-test-slave-9tkh krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:15.531322 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:15.609786 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:15.609999 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:15.610030 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:15.611418 11370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:15.611466 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:15.611485 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:15.611500 11370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:15.611513 11370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:15.613166 11370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:15.613418 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:15.613596 11370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:15.617542 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:15.617869 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:15.618324 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:15.618862 11370 server_base.cc:1034] running on GCE node
I20240501 02:12:15.619112 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:15.619367 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:15.620523 11370 hybrid_clock.cc:648] HybridClock initialized: now 1714529535620501 us; error 34 us; skew 500 ppm
May 01 02:12:15 dist-test-slave-9tkh krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:15.622933 11370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:15.623358 11370 webserver.cc:460] Webserver started at http://127.0.0.1:38633/ using document root <none> and password file <none>
I20240501 02:12:15.623525 11370 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:15.623559 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:15.623642 11370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:15.624599 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "8f26872b5c164c8b948fbd4ebf0d3074"
format_stamp: "Formatted at 2024-05-01 02:12:15 on dist-test-slave-9tkh"
server_key: "c2eb6cd43220b651ed3f8f587aac606c"
server_key_iv: "5a08898ac52893832a9d910e914fb21f"
server_key_version: "encryptionkey@0"
I20240501 02:12:15.624879 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "8f26872b5c164c8b948fbd4ebf0d3074"
format_stamp: "Formatted at 2024-05-01 02:12:15 on dist-test-slave-9tkh"
server_key: "c2eb6cd43220b651ed3f8f587aac606c"
server_key_iv: "5a08898ac52893832a9d910e914fb21f"
server_key_version: "encryptionkey@0"
I20240501 02:12:15.626003 11370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:15.626632 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:15.626785 11370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:15.626859 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "8f26872b5c164c8b948fbd4ebf0d3074"
format_stamp: "Formatted at 2024-05-01 02:12:15 on dist-test-slave-9tkh"
server_key: "c2eb6cd43220b651ed3f8f587aac606c"
server_key_iv: "5a08898ac52893832a9d910e914fb21f"
server_key_version: "encryptionkey@0"
I20240501 02:12:15.626920 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:15.657968 11370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:15.660116 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:15.660238 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:15.663836 11370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44487
I20240501 02:12:15.663885 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44487 every 8 connection(s)
I20240501 02:12:15.664175 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:15.664920 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:15.665948 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11370
I20240501 02:12:15.666038 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:15.666180 22577 external_mini_cluster.cc:1428] Setting key e8c146fe180a9c7bc715a57250864a46
I20240501 02:12:15.667378 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074: Bootstrap starting.
I20240501 02:12:15.667977 11438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:15.668210 11438 log.cc:824] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:15.668784 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074: No bootstrap required, opened a new log
I20240501 02:12:15.670583 11438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44487 } }
I20240501 02:12:15.670706 11438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:15.670728 11438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f26872b5c164c8b948fbd4ebf0d3074, State: Initialized, Role: FOLLOWER
I20240501 02:12:15.670821 11438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44487 } }
I20240501 02:12:15.670888 11438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:15.670917 11438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:15.670964 11438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:15.671458 11438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44487 } }
I20240501 02:12:15.671552 11438 leader_election.cc:302] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f26872b5c164c8b948fbd4ebf0d3074; no voters:
I20240501 02:12:15.671697 11438 leader_election.cc:288] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [CANDIDATE]: Term 1 election: Requested vote from peers
May 01 02:12:15 dist-test-slave-9tkh krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 02:12:15.671744 11443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:15.671911 11443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [term 1 LEADER]: Becoming Leader. State: Replica: 8f26872b5c164c8b948fbd4ebf0d3074, State: Running, Role: LEADER
I20240501 02:12:15.671928 11438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:15.672070 11443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44487 } }
I20240501 02:12:15.672618 11444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f26872b5c164c8b948fbd4ebf0d3074. Latest consensus state: current_term: 1 leader_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44487 } } }
I20240501 02:12:15.672688 11444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:15.673223 11445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f26872b5c164c8b948fbd4ebf0d3074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44487 } } }
I20240501 02:12:15.673341 11445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074 [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:15.674636 11459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:15.674736 11459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:15.674837 11452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:15.675010 11452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:15.675734 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:15.666673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37879 (local address 127.0.0.1:44487)
0501 02:12:15.666857 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:15.666869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:15.666895 (+ 26us) server_negotiation.cc:407] Connection header received
0501 02:12:15.667433 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:15.667442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:15.667503 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:15.667661 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:15.668347 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:15.669111 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:15.669750 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:15.669979 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:15.672275 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:15.672285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:15.672285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:15.672310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:15.674199 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:15.674591 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:15.674594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:15.674594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:15.674644 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:15.674817 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:15.674817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:15.674818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:15.675044 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:15.675205 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:15.675465 (+ 260us) server_negotiation.cc:299] Negotiation successful
0501 02:12:15.675568 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20240501 02:12:15.676367 11452 catalog_manager.cc:1347] Generated new cluster ID: 2d59cf55b2bb4e518286bc4545d439cf
I20240501 02:12:15.676419 11452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:15.676613 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:15.684424 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:15.678817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:53181 (local address 127.0.0.1:44487)
0501 02:12:15.678855 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:15.678857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:15.678943 (+ 86us) server_negotiation.cc:407] Connection header received
0501 02:12:15.678999 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:15.679000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:15.679010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:15.679052 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:15.679670 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:15.680377 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:15.680874 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:15.681343 (+ 469us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:15.681978 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:15.681984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:15.681984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:15.681992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:15.683355 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:15.683734 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:15.683735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:15.683736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:15.683759 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:15.683927 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:15.683928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:15.683928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:15.684016 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:15.684107 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:15.684262 (+ 155us) server_negotiation.cc:299] Negotiation successful
0501 02:12:15.684326 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:15.690774 11452 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:15.691325 11452 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:15.695782 11452 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8f26872b5c164c8b948fbd4ebf0d3074: Generated new TSK 0
I20240501 02:12:15.696009 11452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:15.700695 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11370
W20240501 02:12:15.705070 11355 connection.cc:535] client connection to 127.0.0.1:44487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44487): Connection reset by peer (error 104)
2024-05-01T02:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1169 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:15 dist-test-slave-9tkh krb5kdc[11469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:15 dist-test-slave-9tkh krb5kdc[11469](info): set up 2 sockets
May 01 02:12:15 dist-test-slave-9tkh krb5kdc[11469](info): commencing operation
krb5kdc: starting...
W20240501 02:12:16.590448 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:16 dist-test-slave-9tkh krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:16.701835 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:16.781836 11482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:16.782042 11482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:16.782063 11482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:16.783450 11482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:16.783500 11482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:16.783514 11482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:16.783524 11482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:16.783533 11482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:16.785200 11482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:16.785408 11482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:16.785563 11482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:16.789727 11487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:16.790025 11490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:16.790231 11488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:16.790686 11482 server_base.cc:1034] running on GCE node
I20240501 02:12:16.790939 11482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:16.791182 11482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:16.792322 11482 hybrid_clock.cc:648] HybridClock initialized: now 1714529536792305 us; error 29 us; skew 500 ppm
May 01 02:12:16 dist-test-slave-9tkh krb5kdc[11469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:16.794745 11482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:16.795202 11482 webserver.cc:460] Webserver started at http://127.0.0.1:34133/ using document root <none> and password file <none>
I20240501 02:12:16.795382 11482 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:16.795423 11482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:16.795532 11482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:16.796536 11482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "bf0dd37455aa4e018b485176c50435db"
format_stamp: "Formatted at 2024-05-01 02:12:16 on dist-test-slave-9tkh"
server_key: "b4ac0766be39819beb77777e50991994"
server_key_iv: "5c81cdb2aace81202a5a5ca87a43ee53"
server_key_version: "encryptionkey@0"
I20240501 02:12:16.796849 11482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "bf0dd37455aa4e018b485176c50435db"
format_stamp: "Formatted at 2024-05-01 02:12:16 on dist-test-slave-9tkh"
server_key: "b4ac0766be39819beb77777e50991994"
server_key_iv: "5c81cdb2aace81202a5a5ca87a43ee53"
server_key_version: "encryptionkey@0"
I20240501 02:12:16.798029 11482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:12:16.798702 11497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:16.798862 11482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:16.798939 11482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "bf0dd37455aa4e018b485176c50435db"
format_stamp: "Formatted at 2024-05-01 02:12:16 on dist-test-slave-9tkh"
server_key: "b4ac0766be39819beb77777e50991994"
server_key_iv: "5c81cdb2aace81202a5a5ca87a43ee53"
server_key_version: "encryptionkey@0"
I20240501 02:12:16.799000 11482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:16.810084 11482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:16.812111 11482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:16.812230 11482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:16.815917 11482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35467
I20240501 02:12:16.816004 11549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35467 every 8 connection(s)
I20240501 02:12:16.816278 11482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:16.816846 11550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:16.819052 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db: Bootstrap starting.
I20240501 02:12:16.819612 11550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:16.819851 11550 log.cc:824] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:16.820497 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db: No bootstrap required, opened a new log
I20240501 02:12:16.821916 11550 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0dd37455aa4e018b485176c50435db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20240501 02:12:16.822039 11550 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:16.822062 11550 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0dd37455aa4e018b485176c50435db, State: Initialized, Role: FOLLOWER
I20240501 02:12:16.822176 11550 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0dd37455aa4e018b485176c50435db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20240501 02:12:16.822233 11550 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:16.822268 11550 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:16.822316 11550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:16.822890 11550 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0dd37455aa4e018b485176c50435db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20240501 02:12:16.822983 11550 leader_election.cc:302] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0dd37455aa4e018b485176c50435db; no voters:
I20240501 02:12:16.823127 11550 leader_election.cc:288] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:16.823310 11553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:16.823379 11550 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:16.823462 11553 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [term 1 LEADER]: Becoming Leader. State: Replica: bf0dd37455aa4e018b485176c50435db, State: Running, Role: LEADER
I20240501 02:12:16.823545 11553 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0dd37455aa4e018b485176c50435db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20240501 02:12:16.823858 11554 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0dd37455aa4e018b485176c50435db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0dd37455aa4e018b485176c50435db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } } }
I20240501 02:12:16.823959 11554 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:16.823874 11555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0dd37455aa4e018b485176c50435db. Latest consensus state: current_term: 1 leader_uuid: "bf0dd37455aa4e018b485176c50435db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0dd37455aa4e018b485176c50435db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } } }
I20240501 02:12:16.824138 11555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:16.824245 11559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:16.824759 11559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:16.826216 11559 catalog_manager.cc:1347] Generated new cluster ID: e67a3fe89a114ddc8ce8d3952a2937f3
I20240501 02:12:16.826277 11559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:16.826341 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11482
I20240501 02:12:16.826445 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:16.826642 22577 external_mini_cluster.cc:1428] Setting key 9e862d4c9413abb1c15d5d547ab333be
I20240501 02:12:16.839339 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:16.827134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:49623 (local address 127.0.0.1:35467)
0501 02:12:16.827298 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:16.827309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:16.834852 (+ 7543us) server_negotiation.cc:407] Connection header received
0501 02:12:16.835020 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:16.835027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:16.835087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:16.835248 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:16.835662 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:16.836877 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:16.837714 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:16.837948 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:16.838650 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:16.838660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:16.838660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:16.838864 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:16.838888 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:16.838897 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:16.838997 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:16.839114 (+ 117us) server_negotiation.cc:299] Negotiation successful
0501 02:12:16.839143 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20240501 02:12:16.840319 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:16.845947 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:16.842864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:52091 (local address 127.0.0.1:35467)
0501 02:12:16.842916 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:16.842918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:16.843142 (+ 224us) server_negotiation.cc:407] Connection header received
0501 02:12:16.843202 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:16.843203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:16.843215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:16.843292 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:16.843623 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:16.844299 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:16.845020 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:16.845167 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:16.845690 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:16.845694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:16.845694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:16.845699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:16.845712 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:16.845716 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:16.845773 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:16.845852 (+ 79us) server_negotiation.cc:299] Negotiation successful
0501 02:12:16.845863 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 02:12:16.846710 11559 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:16.847213 11559 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:16.855943 11559 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bf0dd37455aa4e018b485176c50435db: Generated new TSK 0
I20240501 02:12:16.856132 11559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:16.860752 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11482
W20240501 02:12:16.865322 11467 connection.cc:535] client connection to 127.0.0.1:35467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35467): Connection reset by peer (error 104)
2024-05-01T02:12:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1159 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:16 dist-test-slave-9tkh krb5kdc[11581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:16 dist-test-slave-9tkh krb5kdc[11581](info): set up 2 sockets
May 01 02:12:16 dist-test-slave-9tkh krb5kdc[11581](info): commencing operation
krb5kdc: starting...
W20240501 02:12:17.742468 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.864s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:17 dist-test-slave-9tkh krb5kdc[11581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 02:12:17.853946 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:17.932734 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:17.932951 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:17.932982 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:17.934320 11594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:17.934376 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:17.934394 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:17.934409 11594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:17.934422 11594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:17.936121 11594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:17.936350 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:17.936542 11594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:17.940955 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:17.941206 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:17.941434 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:17.941792 11594 server_base.cc:1034] running on GCE node
I20240501 02:12:17.942034 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:17.942281 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:17.943426 11594 hybrid_clock.cc:648] HybridClock initialized: now 1714529537943407 us; error 37 us; skew 500 ppm
May 01 02:12:17 dist-test-slave-9tkh krb5kdc[11581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:17.952576 11594 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 02:12:17.953078 11594 webserver.cc:460] Webserver started at http://127.0.0.1:42241/ using document root <none> and password file <none>
I20240501 02:12:17.953254 11594 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:17.953300 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:17.953408 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:17.954424 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "93d38b81152643d3becbe39511a3abcd"
format_stamp: "Formatted at 2024-05-01 02:12:17 on dist-test-slave-9tkh"
server_key: "ebd029aa998dd1ed119d07eb3807aab7"
server_key_iv: "657baba4673845a285da906afc99c68f"
server_key_version: "encryptionkey@0"
I20240501 02:12:17.954787 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "93d38b81152643d3becbe39511a3abcd"
format_stamp: "Formatted at 2024-05-01 02:12:17 on dist-test-slave-9tkh"
server_key: "ebd029aa998dd1ed119d07eb3807aab7"
server_key_iv: "657baba4673845a285da906afc99c68f"
server_key_version: "encryptionkey@0"
I20240501 02:12:17.955962 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:17.956591 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:17.956796 11594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:17.956890 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "93d38b81152643d3becbe39511a3abcd"
format_stamp: "Formatted at 2024-05-01 02:12:17 on dist-test-slave-9tkh"
server_key: "ebd029aa998dd1ed119d07eb3807aab7"
server_key_iv: "657baba4673845a285da906afc99c68f"
server_key_version: "encryptionkey@0"
I20240501 02:12:17.956951 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:17.960548 11594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:17.962325 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:17.962442 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:17.966485 11594 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39729
I20240501 02:12:17.966523 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39729 every 8 connection(s)
I20240501 02:12:17.966846 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:17.967684 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:17.968283 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11594
I20240501 02:12:17.968384 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:17.968533 22577 external_mini_cluster.cc:1428] Setting key c1fa0380b3a7fbc73bb72dc1122d809d
I20240501 02:12:17.970335 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd: Bootstrap starting.
I20240501 02:12:17.970760 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:17.969009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:33671 (local address 127.0.0.1:39729)
0501 02:12:17.969148 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:17.969157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:17.969177 (+ 20us) server_negotiation.cc:407] Connection header received
0501 02:12:17.969577 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:17.969583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:17.969609 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:17.969729 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:17.970386 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:17.970389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:17.970389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:17.970393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:17.970413 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:17.970420 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:17.970508 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:17.970581 (+ 73us) server_negotiation.cc:299] Negotiation successful
0501 02:12:17.970612 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20240501 02:12:17.970986 11662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:17.971227 11662 log.cc:824] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:17.971587 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:17.971908 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd: No bootstrap required, opened a new log
I20240501 02:12:17.973505 11662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d38b81152643d3becbe39511a3abcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20240501 02:12:17.973623 11662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:17.973639 11662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d38b81152643d3becbe39511a3abcd, State: Initialized, Role: FOLLOWER
I20240501 02:12:17.973718 11662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d38b81152643d3becbe39511a3abcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20240501 02:12:17.973778 11662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:17.973799 11662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:17.973845 11662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:17.974380 11662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d38b81152643d3becbe39511a3abcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20240501 02:12:17.974470 11662 leader_election.cc:302] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d38b81152643d3becbe39511a3abcd; no voters:
I20240501 02:12:17.974619 11662 leader_election.cc:288] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:17.974676 11672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:17.974849 11662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:17.974870 11672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [term 1 LEADER]: Becoming Leader. State: Replica: 93d38b81152643d3becbe39511a3abcd, State: Running, Role: LEADER
I20240501 02:12:17.975025 11672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d38b81152643d3becbe39511a3abcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20240501 02:12:17.975371 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d38b81152643d3becbe39511a3abcd. Latest consensus state: current_term: 1 leader_uuid: "93d38b81152643d3becbe39511a3abcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d38b81152643d3becbe39511a3abcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20240501 02:12:17.975389 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d38b81152643d3becbe39511a3abcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d38b81152643d3becbe39511a3abcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20240501 02:12:17.975574 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:17.975459 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:17.976208 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:17.976658 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:17.977134 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:17.972874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:59117 (local address 127.0.0.1:39729)
0501 02:12:17.976101 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:17.976103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:17.976115 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:12:17.976138 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:17.976139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:17.976144 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:17.976217 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:17.976921 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:17.976922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:17.976922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:17.976927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:17.976944 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:17.976948 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:17.976997 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:17.977058 (+ 61us) server_negotiation.cc:299] Negotiation successful
0501 02:12:17.977071 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3207}
I20240501 02:12:17.978477 11681 catalog_manager.cc:1347] Generated new cluster ID: 00d4162284c94288b45aaf0bc9d93ca7
I20240501 02:12:17.978557 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:17.995139 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:17.995522 11681 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:18.003218 11681 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 93d38b81152643d3becbe39511a3abcd: Generated new TSK 0
I20240501 02:12:18.003417 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:18.011313 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11594
2024-05-01T02:12:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1150 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:18 dist-test-slave-9tkh krb5kdc[11693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:18 dist-test-slave-9tkh krb5kdc[11693](info): set up 2 sockets
May 01 02:12:18 dist-test-slave-9tkh krb5kdc[11693](info): commencing operation
krb5kdc: starting...
W20240501 02:12:18.898418 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:18 dist-test-slave-9tkh krb5kdc[11693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:19.009096 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44197
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:19.088421 11706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:19.088675 11706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:19.088711 11706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:19.090148 11706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:19.090210 11706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:19.090229 11706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:19.090245 11706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:19.090258 11706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:19.092070 11706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:44197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:19.092336 11706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:19.092525 11706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 02:12:19.097035 11706 server_base.cc:1034] running on GCE node
W20240501 02:12:19.097307 11711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:19.097635 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:19.097642 11714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:19.097999 11706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:19.098273 11706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:19.099407 11706 hybrid_clock.cc:648] HybridClock initialized: now 1714529539099388 us; error 34 us; skew 500 ppm
May 01 02:12:19 dist-test-slave-9tkh krb5kdc[11693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529539, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:19.101985 11706 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:19.102437 11706 webserver.cc:460] Webserver started at http://127.22.12.126:40801/ using document root <none> and password file <none>
I20240501 02:12:19.102629 11706 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:19.102671 11706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:19.102751 11706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:19.103729 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "ca56665b8de84f3f958ef2e81021b983"
format_stamp: "Formatted at 2024-05-01 02:12:19 on dist-test-slave-9tkh"
server_key: "f32acea405a81f1a5191052644f14a6a"
server_key_iv: "be762571e08ab8f3e1c93504768955b1"
server_key_version: "encryptionkey@0"
I20240501 02:12:19.104000 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "ca56665b8de84f3f958ef2e81021b983"
format_stamp: "Formatted at 2024-05-01 02:12:19 on dist-test-slave-9tkh"
server_key: "f32acea405a81f1a5191052644f14a6a"
server_key_iv: "be762571e08ab8f3e1c93504768955b1"
server_key_version: "encryptionkey@0"
I20240501 02:12:19.105158 11706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:19.105922 11721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:19.106114 11706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:19.106192 11706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "ca56665b8de84f3f958ef2e81021b983"
format_stamp: "Formatted at 2024-05-01 02:12:19 on dist-test-slave-9tkh"
server_key: "f32acea405a81f1a5191052644f14a6a"
server_key_iv: "be762571e08ab8f3e1c93504768955b1"
server_key_version: "encryptionkey@0"
I20240501 02:12:19.106247 11706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:19.116318 11706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:19.118849 11706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:19.118954 11706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:19.122426 11706 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:44197
I20240501 02:12:19.122457 11773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:44197 every 8 connection(s)
I20240501 02:12:19.122803 11706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:19.123558 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:19.123620 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11706
I20240501 02:12:19.123725 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:19.123908 22577 external_mini_cluster.cc:1428] Setting key d900e48e2f8235307bbb2f0c6edb6040
I20240501 02:12:19.125855 11774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983: Bootstrap starting.
I20240501 02:12:19.126488 11774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:19.126792 11774 log.cc:824] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:19.127527 11774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983: No bootstrap required, opened a new log
I20240501 02:12:19.129427 11774 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca56665b8de84f3f958ef2e81021b983" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44197 } }
May 01 02:12:19 dist-test-slave-9tkh krb5kdc[11693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:19.129592 11774 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:19.129616 11774 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca56665b8de84f3f958ef2e81021b983, State: Initialized, Role: FOLLOWER
I20240501 02:12:19.129724 11774 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca56665b8de84f3f958ef2e81021b983" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44197 } }
I20240501 02:12:19.129798 11774 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:19.129834 11774 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:19.129884 11774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:19.130573 11774 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca56665b8de84f3f958ef2e81021b983" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44197 } }
I20240501 02:12:19.130666 11774 leader_election.cc:302] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca56665b8de84f3f958ef2e81021b983; no voters:
I20240501 02:12:19.130810 11774 leader_election.cc:288] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:19.130934 11779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:19.131062 11774 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:19.131117 11779 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [term 1 LEADER]: Becoming Leader. State: Replica: ca56665b8de84f3f958ef2e81021b983, State: Running, Role: LEADER
I20240501 02:12:19.131208 11779 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca56665b8de84f3f958ef2e81021b983" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44197 } }
I20240501 02:12:19.131492 11779 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca56665b8de84f3f958ef2e81021b983. Latest consensus state: current_term: 1 leader_uuid: "ca56665b8de84f3f958ef2e81021b983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca56665b8de84f3f958ef2e81021b983" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44197 } } }
I20240501 02:12:19.131567 11779 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:19.132172 11780 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca56665b8de84f3f958ef2e81021b983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca56665b8de84f3f958ef2e81021b983" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44197 } } }
I20240501 02:12:19.132246 11780 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983 [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:19.132962 11794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:19.133013 11794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:19.133076 11787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:19.133232 11787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:19.134609 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:19.124410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54433 (local address 127.22.12.126:44197)
0501 02:12:19.124604 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:19.124617 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:19.124635 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:12:19.125056 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:19.125062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:19.125105 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:19.125234 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:19.125982 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:19.127102 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:19.127751 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:19.127992 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:19.130178 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:19.130187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:19.130188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:19.130206 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:19.133016 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:19.133498 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:19.133501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:19.133501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:19.133564 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:19.133764 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:19.133765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:19.133765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:19.133966 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:19.134121 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:19.134281 (+ 160us) server_negotiation.cc:299] Negotiation successful
0501 02:12:19.134391 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20240501 02:12:19.135124 11787 catalog_manager.cc:1347] Generated new cluster ID: 46153ae122c5480a97c799cb7e0691f0
I20240501 02:12:19.135166 11787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:19.135612 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:19.143565 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:19.136869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50119 (local address 127.22.12.126:44197)
0501 02:12:19.136900 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:19.136902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:19.137011 (+ 109us) server_negotiation.cc:407] Connection header received
0501 02:12:19.137088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:19.137089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:19.137097 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:19.137122 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:19.137901 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:19.138876 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:19.139621 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:19.139781 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:19.140903 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:19.140906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:19.140907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:19.140913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:19.142489 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:19.142887 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:19.142888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:19.142889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:19.142911 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:19.143083 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:19.143083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:19.143084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:19.143183 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:19.143269 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:19.143406 (+ 137us) server_negotiation.cc:299] Negotiation successful
0501 02:12:19.143467 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:12:19.150697 11787 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:19.151489 11787 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:19.163981 11787 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ca56665b8de84f3f958ef2e81021b983: Generated new TSK 0
I20240501 02:12:19.164225 11787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:19.167508 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11706
W20240501 02:12:19.171823 11691 connection.cc:535] client connection to 127.22.12.126:44197 recv error: Network error: failed to read from TLS socket (remote: 127.22.12.126:44197): Connection reset by peer (error 104)
2024-05-01T02:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:19 dist-test-slave-9tkh krb5kdc[11805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:19 dist-test-slave-9tkh krb5kdc[11805](info): set up 2 sockets
May 01 02:12:19 dist-test-slave-9tkh krb5kdc[11805](info): commencing operation
krb5kdc: starting...
W20240501 02:12:20.066450 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:20 dist-test-slave-9tkh krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:20.179105 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41247
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:20.258322 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:20.258560 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:20.258582 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:20.259909 11818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:20.259953 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:20.259964 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:20.259975 11818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:20.259985 11818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:20.261605 11818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:20.261785 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:20.261936 11818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:20.266012 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:20.266135 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:20.266491 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:20.266881 11818 server_base.cc:1034] running on GCE node
I20240501 02:12:20.267124 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:20.267355 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:20.268484 11818 hybrid_clock.cc:648] HybridClock initialized: now 1714529540268463 us; error 37 us; skew 500 ppm
May 01 02:12:20 dist-test-slave-9tkh krb5kdc[11805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529540, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:20.270784 11818 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:20.271195 11818 webserver.cc:460] Webserver started at http://127.22.12.126:38941/ using document root <none> and password file <none>
I20240501 02:12:20.271363 11818 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:20.271401 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:20.271498 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:20.272485 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "590efd509f544b5d935e0f1e8a1ecfc7"
format_stamp: "Formatted at 2024-05-01 02:12:20 on dist-test-slave-9tkh"
server_key: "d8d9aa94f376decf5e93de4d850deda3"
server_key_iv: "6b3a2c449b63f0677b329b61b1465814"
server_key_version: "encryptionkey@0"
I20240501 02:12:20.272783 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "590efd509f544b5d935e0f1e8a1ecfc7"
format_stamp: "Formatted at 2024-05-01 02:12:20 on dist-test-slave-9tkh"
server_key: "d8d9aa94f376decf5e93de4d850deda3"
server_key_iv: "6b3a2c449b63f0677b329b61b1465814"
server_key_version: "encryptionkey@0"
I20240501 02:12:20.273953 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:12:20.274605 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:20.274775 11818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:20.274849 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "590efd509f544b5d935e0f1e8a1ecfc7"
format_stamp: "Formatted at 2024-05-01 02:12:20 on dist-test-slave-9tkh"
server_key: "d8d9aa94f376decf5e93de4d850deda3"
server_key_iv: "6b3a2c449b63f0677b329b61b1465814"
server_key_version: "encryptionkey@0"
I20240501 02:12:20.274902 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:20.288281 11818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:20.290347 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:20.290460 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:20.294099 11818 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41247
I20240501 02:12:20.294150 11885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41247 every 8 connection(s)
I20240501 02:12:20.294440 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:20.295248 11886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:20.297356 11886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7: Bootstrap starting.
I20240501 02:12:20.297912 11886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:20.298146 11886 log.cc:824] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:20.298789 11886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7: No bootstrap required, opened a new log
I20240501 02:12:20.300194 11886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41247 } }
I20240501 02:12:20.300319 11886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:20.300341 11886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590efd509f544b5d935e0f1e8a1ecfc7, State: Initialized, Role: FOLLOWER
I20240501 02:12:20.300458 11886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41247 } }
I20240501 02:12:20.300516 11886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:20.300544 11886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:20.300590 11886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:20.301113 11886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41247 } }
I20240501 02:12:20.301208 11886 leader_election.cc:302] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 590efd509f544b5d935e0f1e8a1ecfc7; no voters:
I20240501 02:12:20.301362 11886 leader_election.cc:288] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:20.301405 11889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:20.301570 11889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 590efd509f544b5d935e0f1e8a1ecfc7, State: Running, Role: LEADER
I20240501 02:12:20.301651 11889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41247 } }
I20240501 02:12:20.301888 11886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:20.301987 11891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 590efd509f544b5d935e0f1e8a1ecfc7. Latest consensus state: current_term: 1 leader_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41247 } } }
I20240501 02:12:20.302001 11890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "590efd509f544b5d935e0f1e8a1ecfc7" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41247 } } }
I20240501 02:12:20.302073 11891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:20.302089 11890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7 [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:20.302273 11893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:20.302770 11893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:20.303830 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11818
I20240501 02:12:20.303941 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:20.304114 22577 external_mini_cluster.cc:1428] Setting key f2f380bed95cf4e574b9f467af27c789
I20240501 02:12:20.304411 11893 catalog_manager.cc:1347] Generated new cluster ID: f01fdeed522a4b3eba671c8b6dad83d4
I20240501 02:12:20.304467 11893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:20.309167 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:20.304712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54839 (local address 127.22.12.126:41247)
0501 02:12:20.304906 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:20.304918 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:20.304940 (+ 22us) server_negotiation.cc:407] Connection header received
0501 02:12:20.305041 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:20.305047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:20.305108 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:20.305242 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:20.305660 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:20.306810 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:20.307638 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:20.307875 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:20.308510 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:20.308516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:20.308517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:20.308716 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:20.308744 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:20.308752 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:20.308849 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:20.308962 (+ 113us) server_negotiation.cc:299] Negotiation successful
0501 02:12:20.308990 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":112,"threads_started":1}
I20240501 02:12:20.309862 11893 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:20.310158 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:20.310632 11893 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:20.316396 11893 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 590efd509f544b5d935e0f1e8a1ecfc7: Generated new TSK 0
I20240501 02:12:20.316613 11893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:20.317823 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:20.314822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60537 (local address 127.22.12.126:41247)
0501 02:12:20.314867 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:20.314869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:20.315061 (+ 192us) server_negotiation.cc:407] Connection header received
0501 02:12:20.315117 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:20.315118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:20.315128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:20.315181 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:20.315501 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:20.316283 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:20.316961 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:20.317144 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:20.317560 (+ 416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:20.317563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:20.317564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:20.317568 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:20.317582 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:20.317586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:20.317668 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:20.317739 (+ 71us) server_negotiation.cc:299] Negotiation successful
0501 02:12:20.317749 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:12:20.319811 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11818
W20240501 02:12:20.324033 11803 connection.cc:535] client connection to 127.22.12.126:41247 recv error: Network error: failed to read from TLS socket (remote: 127.22.12.126:41247): Connection reset by peer (error 104)
2024-05-01T02:12:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1152 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:20 dist-test-slave-9tkh krb5kdc[11917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:20 dist-test-slave-9tkh krb5kdc[11917](info): set up 2 sockets
May 01 02:12:20 dist-test-slave-9tkh krb5kdc[11917](info): commencing operation
krb5kdc: starting...
W20240501 02:12:21.218407 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:21 dist-test-slave-9tkh krb5kdc[11917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:21.330783 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39821
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:21.410288 11930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:21.410495 11930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:21.410545 11930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:21.411906 11930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:21.411953 11930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:21.411971 11930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:21.411985 11930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:21.411998 11930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:21.413642 11930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:39821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:21.413859 11930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:21.414047 11930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:21.417943 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:21.418236 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:21.418663 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:21.418993 11930 server_base.cc:1034] running on GCE node
I20240501 02:12:21.419207 11930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:21.419432 11930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:21.420552 11930 hybrid_clock.cc:648] HybridClock initialized: now 1714529541420537 us; error 33 us; skew 500 ppm
May 01 02:12:21 dist-test-slave-9tkh krb5kdc[11917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529541, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:21.422897 11930 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:21.423301 11930 webserver.cc:460] Webserver started at http://127.22.12.126:34521/ using document root <none> and password file <none>
I20240501 02:12:21.423461 11930 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:21.423498 11930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:21.423583 11930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:21.424538 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "5eed8acd4c2340898621174dafc05d8a"
format_stamp: "Formatted at 2024-05-01 02:12:21 on dist-test-slave-9tkh"
server_key: "b575a66b5e23ce5b038d3da6df8c1493"
server_key_iv: "3e09c66789109ad7171131a8cf9a8111"
server_key_version: "encryptionkey@0"
I20240501 02:12:21.424836 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "5eed8acd4c2340898621174dafc05d8a"
format_stamp: "Formatted at 2024-05-01 02:12:21 on dist-test-slave-9tkh"
server_key: "b575a66b5e23ce5b038d3da6df8c1493"
server_key_iv: "3e09c66789109ad7171131a8cf9a8111"
server_key_version: "encryptionkey@0"
I20240501 02:12:21.425976 11930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:12:21.426707 11945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:21.426883 11930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:12:21.426960 11930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "5eed8acd4c2340898621174dafc05d8a"
format_stamp: "Formatted at 2024-05-01 02:12:21 on dist-test-slave-9tkh"
server_key: "b575a66b5e23ce5b038d3da6df8c1493"
server_key_iv: "3e09c66789109ad7171131a8cf9a8111"
server_key_version: "encryptionkey@0"
I20240501 02:12:21.427019 11930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:21.430372 11930 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:21.432075 11930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:21.432176 11930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:21.435761 11930 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:39821
I20240501 02:12:21.435794 11997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:39821 every 8 connection(s)
I20240501 02:12:21.436075 11930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:21.436895 11998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:21.439021 11998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a: Bootstrap starting.
I20240501 02:12:21.439559 11998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:21.439862 11998 log.cc:824] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:21.440508 11998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a: No bootstrap required, opened a new log
I20240501 02:12:21.441918 11998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed8acd4c2340898621174dafc05d8a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39821 } }
I20240501 02:12:21.442042 11998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:21.442057 11998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eed8acd4c2340898621174dafc05d8a, State: Initialized, Role: FOLLOWER
I20240501 02:12:21.442191 11998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed8acd4c2340898621174dafc05d8a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39821 } }
I20240501 02:12:21.442247 11998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:21.442266 11998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:21.442307 11998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:21.442920 11998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed8acd4c2340898621174dafc05d8a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39821 } }
I20240501 02:12:21.443010 11998 leader_election.cc:302] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eed8acd4c2340898621174dafc05d8a; no voters:
I20240501 02:12:21.443161 11998 leader_election.cc:288] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:21.443214 12001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:21.443349 12001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [term 1 LEADER]: Becoming Leader. State: Replica: 5eed8acd4c2340898621174dafc05d8a, State: Running, Role: LEADER
I20240501 02:12:21.443383 11998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:21.443434 12001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed8acd4c2340898621174dafc05d8a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39821 } }
I20240501 02:12:21.443773 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eed8acd4c2340898621174dafc05d8a. Latest consensus state: current_term: 1 leader_uuid: "5eed8acd4c2340898621174dafc05d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed8acd4c2340898621174dafc05d8a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39821 } } }
I20240501 02:12:21.443861 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:21.443877 12002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eed8acd4c2340898621174dafc05d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed8acd4c2340898621174dafc05d8a" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 39821 } } }
I20240501 02:12:21.443921 12002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:21.444067 12008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:21.444615 12008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:21.445362 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 11930
I20240501 02:12:21.445474 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:21.445644 22577 external_mini_cluster.cc:1428] Setting key 9f5f8c417409e47129a7178cf5a63eb9
I20240501 02:12:21.446187 12008 catalog_manager.cc:1347] Generated new cluster ID: 16ebd5bd642d45f7b8270d59a24b366f
I20240501 02:12:21.446241 12008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:21.448333 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:21.446064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60949 (local address 127.22.12.126:39821)
0501 02:12:21.446477 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:21.446488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:21.446508 (+ 20us) server_negotiation.cc:407] Connection header received
0501 02:12:21.446643 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:21.446648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:21.446677 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:21.446816 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:21.447844 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:21.447847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:21.447848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:21.447852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:21.447877 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:21.447884 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:21.447959 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:21.448134 (+ 175us) server_negotiation.cc:299] Negotiation successful
0501 02:12:21.448168 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":73,"threads_started":1}
I20240501 02:12:21.449229 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:21.451114 12008 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:21.451254 12008 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:21.451742 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:21.450708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47725 (local address 127.22.12.126:39821)
0501 02:12:21.450741 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:21.450743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:21.450861 (+ 118us) server_negotiation.cc:407] Connection header received
0501 02:12:21.450936 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:21.450937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:21.450944 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:21.451001 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:21.451554 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:21.451555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:21.451556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:21.451559 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:21.451573 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:21.451577 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:21.451622 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:21.451669 (+ 47us) server_negotiation.cc:299] Negotiation successful
0501 02:12:21.451683 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:12:21.465073 12008 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5eed8acd4c2340898621174dafc05d8a: Generated new TSK 0
I20240501 02:12:21.465297 12008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:21.468079 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 11930
2024-05-01T02:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1148 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:21 dist-test-slave-9tkh krb5kdc[12029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:21 dist-test-slave-9tkh krb5kdc[12029](info): set up 2 sockets
May 01 02:12:21 dist-test-slave-9tkh krb5kdc[12029](info): commencing operation
krb5kdc: starting...
W20240501 02:12:22.358636 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:22 dist-test-slave-9tkh krb5kdc[12029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:22.468673 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41947
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:41947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:22.546835 12042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:22.547066 12042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:22.547101 12042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:22.548471 12042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:22.548524 12042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:22.548542 12042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:22.548556 12042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:22.548570 12042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:22.550307 12042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:41947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:41947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:22.550570 12042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:22.550761 12042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:22.554855 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:22.555176 12047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:22.555387 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:22.555778 12042 server_base.cc:1034] running on GCE node
I20240501 02:12:22.555989 12042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:22.556195 12042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:22.557340 12042 hybrid_clock.cc:648] HybridClock initialized: now 1714529542557325 us; error 28 us; skew 500 ppm
May 01 02:12:22 dist-test-slave-9tkh krb5kdc[12029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529542, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:22.559784 12042 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:22.560171 12042 webserver.cc:460] Webserver started at http://127.22.12.126:46495/ using document root <none> and password file <none>
I20240501 02:12:22.560319 12042 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:22.560359 12042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:22.560436 12042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:22.561379 12042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "c22661f642944523a8530d8a2b3ee9df"
format_stamp: "Formatted at 2024-05-01 02:12:22 on dist-test-slave-9tkh"
server_key: "a19618baa5e775bbb7b36783e7c9e3f6"
server_key_iv: "a2f18a1372a71c77a7e37b3b23af3635"
server_key_version: "encryptionkey@0"
I20240501 02:12:22.561630 12042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "c22661f642944523a8530d8a2b3ee9df"
format_stamp: "Formatted at 2024-05-01 02:12:22 on dist-test-slave-9tkh"
server_key: "a19618baa5e775bbb7b36783e7c9e3f6"
server_key_iv: "a2f18a1372a71c77a7e37b3b23af3635"
server_key_version: "encryptionkey@0"
I20240501 02:12:22.562727 12042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:22.563356 12057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:22.563519 12042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:22.563597 12042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "c22661f642944523a8530d8a2b3ee9df"
format_stamp: "Formatted at 2024-05-01 02:12:22 on dist-test-slave-9tkh"
server_key: "a19618baa5e775bbb7b36783e7c9e3f6"
server_key_iv: "a2f18a1372a71c77a7e37b3b23af3635"
server_key_version: "encryptionkey@0"
I20240501 02:12:22.563652 12042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:22.578161 12042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:22.580154 12042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:22.580262 12042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:22.583655 12042 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:41947
I20240501 02:12:22.583703 12109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:41947 every 8 connection(s)
I20240501 02:12:22.584012 12042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:22.584761 12110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:22.586951 12110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df: Bootstrap starting.
I20240501 02:12:22.587503 12110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:22.587736 12110 log.cc:824] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:22.588394 12110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df: No bootstrap required, opened a new log
I20240501 02:12:22.589797 12110 raft_consensus.cc:348] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c22661f642944523a8530d8a2b3ee9df" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41947 } }
I20240501 02:12:22.589922 12110 raft_consensus.cc:374] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:22.589946 12110 raft_consensus.cc:729] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22661f642944523a8530d8a2b3ee9df, State: Initialized, Role: FOLLOWER
I20240501 02:12:22.590060 12110 consensus_queue.cc:260] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c22661f642944523a8530d8a2b3ee9df" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41947 } }
I20240501 02:12:22.590118 12110 raft_consensus.cc:388] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:22.590147 12110 raft_consensus.cc:482] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:22.590191 12110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:22.590766 12110 raft_consensus.cc:504] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c22661f642944523a8530d8a2b3ee9df" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41947 } }
I20240501 02:12:22.590865 12110 leader_election.cc:302] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c22661f642944523a8530d8a2b3ee9df; no voters:
I20240501 02:12:22.591014 12110 leader_election.cc:288] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:22.591069 12113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:22.591235 12113 raft_consensus.cc:686] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [term 1 LEADER]: Becoming Leader. State: Replica: c22661f642944523a8530d8a2b3ee9df, State: Running, Role: LEADER
I20240501 02:12:22.591320 12113 consensus_queue.cc:237] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c22661f642944523a8530d8a2b3ee9df" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41947 } }
I20240501 02:12:22.591540 12110 sys_catalog.cc:564] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:22.591682 12115 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [sys.catalog]: SysCatalogTable state changed. Reason: New leader c22661f642944523a8530d8a2b3ee9df. Latest consensus state: current_term: 1 leader_uuid: "c22661f642944523a8530d8a2b3ee9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c22661f642944523a8530d8a2b3ee9df" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41947 } } }
I20240501 02:12:22.591712 12114 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c22661f642944523a8530d8a2b3ee9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c22661f642944523a8530d8a2b3ee9df" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 41947 } } }
I20240501 02:12:22.591756 12115 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:22.591768 12114 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:22.591936 12117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:22.592399 12117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:22.593230 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12042
I20240501 02:12:22.593327 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:22.593477 22577 external_mini_cluster.cc:1428] Setting key 8bbc32908fcd5f919d994da9cde3c9dc
I20240501 02:12:22.593941 12117 catalog_manager.cc:1347] Generated new cluster ID: eae6d78dee9041aeb768f8bf8c2cd03e
I20240501 02:12:22.594036 12117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:12:22 dist-test-slave-9tkh krb5kdc[12029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:22.601627 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:22.594127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39153 (local address 127.22.12.126:41947)
0501 02:12:22.594270 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:22.594279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:22.594296 (+ 17us) server_negotiation.cc:407] Connection header received
0501 02:12:22.594382 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:22.594387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:22.594423 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:22.594560 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:22.595272 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:22.595986 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:22.596555 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:22.596797 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:22.598783 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:22.598788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:22.598789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:22.598804 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:22.600391 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:22.600686 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:22.600689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:22.600689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:22.600737 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:22.600876 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:22.600877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:22.600877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:22.601077 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:22.601197 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:22.601310 (+ 113us) server_negotiation.cc:299] Negotiation successful
0501 02:12:22.601435 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20240501 02:12:22.602519 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:22.611807 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:22.606825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42067 (local address 127.22.12.126:41947)
0501 02:12:22.606872 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:22.606874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:22.606980 (+ 106us) server_negotiation.cc:407] Connection header received
0501 02:12:22.607034 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:22.607035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:22.607047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:22.607094 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:22.607687 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:22.608115 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:22.608599 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:22.608778 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:22.609669 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:22.609672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:22.609672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:22.609678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:22.610947 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:22.611222 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:22.611223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:22.611223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:22.611244 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:22.611365 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:22.611365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:22.611366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:22.611448 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:22.611550 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:22.611635 (+ 85us) server_negotiation.cc:299] Negotiation successful
0501 02:12:22.611713 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:12:22.624526 12117 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:22.625427 12117 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:22.633108 12117 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c22661f642944523a8530d8a2b3ee9df: Generated new TSK 0
I20240501 02:12:22.633340 12117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:22.643239 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12042
2024-05-01T02:12:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:22 dist-test-slave-9tkh krb5kdc[12141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:22 dist-test-slave-9tkh krb5kdc[12141](info): set up 2 sockets
May 01 02:12:22 dist-test-slave-9tkh krb5kdc[12141](info): commencing operation
krb5kdc: starting...
W20240501 02:12:23.538453 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:23 dist-test-slave-9tkh krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:23.649520 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:37177
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:23.728866 12154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:23.729086 12154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:23.729108 12154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:23.730520 12154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:23.730603 12154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:23.730616 12154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:23.730628 12154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:23.730638 12154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:23.732378 12154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:37177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:23.732621 12154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:23.732834 12154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:23.736776 12160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:23.737190 12159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:23.737507 12162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:23.737851 12154 server_base.cc:1034] running on GCE node
I20240501 02:12:23.738072 12154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:23.738296 12154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:23.739447 12154 hybrid_clock.cc:648] HybridClock initialized: now 1714529543739430 us; error 36 us; skew 500 ppm
May 01 02:12:23 dist-test-slave-9tkh krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529543, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:23.745887 12154 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:23.746351 12154 webserver.cc:460] Webserver started at http://127.22.12.126:33989/ using document root <none> and password file <none>
I20240501 02:12:23.746560 12154 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:23.746613 12154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:23.746722 12154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:23.747865 12154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "d2121158823b428fb9762349ff0293cc"
format_stamp: "Formatted at 2024-05-01 02:12:23 on dist-test-slave-9tkh"
server_key: "f184bf7f828341a1664d092a2019da98"
server_key_iv: "2c41476531623536b0563fcd1a962438"
server_key_version: "encryptionkey@0"
I20240501 02:12:23.748194 12154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "d2121158823b428fb9762349ff0293cc"
format_stamp: "Formatted at 2024-05-01 02:12:23 on dist-test-slave-9tkh"
server_key: "f184bf7f828341a1664d092a2019da98"
server_key_iv: "2c41476531623536b0563fcd1a962438"
server_key_version: "encryptionkey@0"
I20240501 02:12:23.749433 12154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240501 02:12:23.750167 12169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:23.750329 12154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:23.750419 12154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "d2121158823b428fb9762349ff0293cc"
format_stamp: "Formatted at 2024-05-01 02:12:23 on dist-test-slave-9tkh"
server_key: "f184bf7f828341a1664d092a2019da98"
server_key_iv: "2c41476531623536b0563fcd1a962438"
server_key_version: "encryptionkey@0"
I20240501 02:12:23.750478 12154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:23.757798 12154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:23.759608 12154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:23.759719 12154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:23.763301 12154 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:37177
I20240501 02:12:23.763337 12221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:37177 every 8 connection(s)
I20240501 02:12:23.763636 12154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:23.763983 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12154
I20240501 02:12:23.764101 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:23.764279 22577 external_mini_cluster.cc:1428] Setting key dbae9555a8a96b8b4c6723000a33f0b2
I20240501 02:12:23.764518 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:23.767401 12222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc: Bootstrap starting.
I20240501 02:12:23.768021 12222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:23.768271 12222 log.cc:824] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:23.768862 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:23.764753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58561 (local address 127.22.12.126:37177)
0501 02:12:23.764914 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:23.764922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:23.764941 (+ 19us) server_negotiation.cc:407] Connection header received
0501 02:12:23.765530 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:23.765535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:23.765576 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:23.765688 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:23.766100 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:23.766868 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:23.767661 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:23.767841 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:23.768293 (+ 452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:23.768300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:23.768300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:23.768501 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:23.768518 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:23.768526 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:23.768583 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:23.768673 (+ 90us) server_negotiation.cc:299] Negotiation successful
0501 02:12:23.768710 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20240501 02:12:23.769032 12222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc: No bootstrap required, opened a new log
I20240501 02:12:23.770002 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:23.773816 12222 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2121158823b428fb9762349ff0293cc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37177 } }
I20240501 02:12:23.773989 12222 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:23.774017 12222 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2121158823b428fb9762349ff0293cc, State: Initialized, Role: FOLLOWER
I20240501 02:12:23.774137 12222 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2121158823b428fb9762349ff0293cc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37177 } }
I20240501 02:12:23.774219 12222 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:23.774259 12222 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:23.774322 12222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:23.775043 12222 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2121158823b428fb9762349ff0293cc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37177 } }
I20240501 02:12:23.775141 12222 leader_election.cc:302] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2121158823b428fb9762349ff0293cc; no voters:
I20240501 02:12:23.775295 12222 leader_election.cc:288] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:23.775349 12232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:23.775540 12222 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:23.775527 12232 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [term 1 LEADER]: Becoming Leader. State: Replica: d2121158823b428fb9762349ff0293cc, State: Running, Role: LEADER
I20240501 02:12:23.775724 12232 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2121158823b428fb9762349ff0293cc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37177 } }
I20240501 02:12:23.776038 12232 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2121158823b428fb9762349ff0293cc. Latest consensus state: current_term: 1 leader_uuid: "d2121158823b428fb9762349ff0293cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2121158823b428fb9762349ff0293cc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37177 } } }
I20240501 02:12:23.776166 12232 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:23.776391 12241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:23.776888 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:23.773988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45921 (local address 127.22.12.126:37177)
0501 02:12:23.774068 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:23.774070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:23.774207 (+ 137us) server_negotiation.cc:407] Connection header received
0501 02:12:23.774303 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:23.774305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:23.774315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:23.774377 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:23.774729 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:23.775227 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:23.775894 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:23.776465 (+ 571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:23.776527 (+ 62us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:23.776530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:23.776531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:23.776535 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:23.776545 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:23.776557 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:23.776609 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:23.776791 (+ 182us) server_negotiation.cc:299] Negotiation successful
0501 02:12:23.776819 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 02:12:23.777000 12241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:23.777307 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2121158823b428fb9762349ff0293cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2121158823b428fb9762349ff0293cc" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 37177 } } }
I20240501 02:12:23.777513 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:23.778586 12241 catalog_manager.cc:1347] Generated new cluster ID: d3b5121cf8e44e259bcdd144e5977945
I20240501 02:12:23.778632 12241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:23.801077 12241 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:23.801878 12241 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:23.814851 12241 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2121158823b428fb9762349ff0293cc: Generated new TSK 0
I20240501 02:12:23.815104 12241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:23.817536 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12154
2024-05-01T02:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:23 dist-test-slave-9tkh krb5kdc[12253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:23 dist-test-slave-9tkh krb5kdc[12253](info): set up 2 sockets
May 01 02:12:23 dist-test-slave-9tkh krb5kdc[12253](info): commencing operation
krb5kdc: starting...
W20240501 02:12:24.706595 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:24 dist-test-slave-9tkh krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:24.817493 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44411
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:44411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:24.895932 12266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:24.896133 12266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:24.896163 12266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:24.897468 12266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:24.897511 12266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:24.897531 12266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:24.897544 12266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:24.897559 12266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:24.899255 12266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:44411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:24.899487 12266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:24.899664 12266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:24.903869 12272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:24.904228 12271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:24.904320 12274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:24.904778 12266 server_base.cc:1034] running on GCE node
I20240501 02:12:24.905019 12266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:24.905247 12266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:24.906383 12266 hybrid_clock.cc:648] HybridClock initialized: now 1714529544906362 us; error 39 us; skew 500 ppm
May 01 02:12:24 dist-test-slave-9tkh krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529544, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:24.908720 12266 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:24.909132 12266 webserver.cc:460] Webserver started at http://127.22.12.126:35059/ using document root <none> and password file <none>
I20240501 02:12:24.909292 12266 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:24.909332 12266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:24.909430 12266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:24.910384 12266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "f0b26f0d908c4619b55623fa0a19855e"
format_stamp: "Formatted at 2024-05-01 02:12:24 on dist-test-slave-9tkh"
server_key: "3fc94e73970c473ebde8d0ac2b265573"
server_key_iv: "4b68e9d2a28ee03ad7f898de32830254"
server_key_version: "encryptionkey@0"
I20240501 02:12:24.910697 12266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "f0b26f0d908c4619b55623fa0a19855e"
format_stamp: "Formatted at 2024-05-01 02:12:24 on dist-test-slave-9tkh"
server_key: "3fc94e73970c473ebde8d0ac2b265573"
server_key_iv: "4b68e9d2a28ee03ad7f898de32830254"
server_key_version: "encryptionkey@0"
I20240501 02:12:24.911808 12266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:24.912433 12281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:24.912600 12266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 02:12:24.912671 12266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "f0b26f0d908c4619b55623fa0a19855e"
format_stamp: "Formatted at 2024-05-01 02:12:24 on dist-test-slave-9tkh"
server_key: "3fc94e73970c473ebde8d0ac2b265573"
server_key_iv: "4b68e9d2a28ee03ad7f898de32830254"
server_key_version: "encryptionkey@0"
I20240501 02:12:24.912729 12266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:24.916133 12266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:24.917871 12266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:24.917982 12266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:24.921531 12266 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:44411
I20240501 02:12:24.921631 12333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:44411 every 8 connection(s)
I20240501 02:12:24.921855 12266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:24.922711 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:24.924796 12334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e: Bootstrap starting.
I20240501 02:12:24.925474 12334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:24.925710 12334 log.cc:824] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:24.926386 12334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e: No bootstrap required, opened a new log
I20240501 02:12:24.927806 12334 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b26f0d908c4619b55623fa0a19855e" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44411 } }
I20240501 02:12:24.927932 12334 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 02:12:24.927948 12334 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b26f0d908c4619b55623fa0a19855e, State: Initialized, Role: FOLLOWER
I20240501 02:12:24.928073 12334 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b26f0d908c4619b55623fa0a19855e" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44411 } }
I20240501 02:12:24.928129 12334 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 02:12:24.928150 12334 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 02:12:24.928191 12334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:24.928718 12334 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b26f0d908c4619b55623fa0a19855e" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44411 } }
I20240501 02:12:24.928807 12334 leader_election.cc:302] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0b26f0d908c4619b55623fa0a19855e; no voters:
I20240501 02:12:24.928965 12334 leader_election.cc:288] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 02:12:24.929024 12337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:24.929176 12337 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [term 1 LEADER]: Becoming Leader. State: Replica: f0b26f0d908c4619b55623fa0a19855e, State: Running, Role: LEADER
I20240501 02:12:24.929183 12334 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:24.929281 12337 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b26f0d908c4619b55623fa0a19855e" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44411 } }
I20240501 02:12:24.929641 12339 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0b26f0d908c4619b55623fa0a19855e. Latest consensus state: current_term: 1 leader_uuid: "f0b26f0d908c4619b55623fa0a19855e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b26f0d908c4619b55623fa0a19855e" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44411 } } }
I20240501 02:12:24.929740 12338 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0b26f0d908c4619b55623fa0a19855e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b26f0d908c4619b55623fa0a19855e" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 44411 } } }
I20240501 02:12:24.929780 12339 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:24.929929 12338 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:24.930187 12346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:24.930745 12346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:24.931913 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12266
I20240501 02:12:24.932011 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:24.932180 22577 external_mini_cluster.cc:1428] Setting key 15e36459bd266d1497c2fa86010c7f59
I20240501 02:12:24.932225 12346 catalog_manager.cc:1347] Generated new cluster ID: a4a36e1026914cf8ae220d16ed9cdaf7
I20240501 02:12:24.932276 12346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:24.940433 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:24.932626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33967 (local address 127.22.12.126:44411)
0501 02:12:24.938865 (+ 6239us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:24.938875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:24.938900 (+ 25us) server_negotiation.cc:407] Connection header received
0501 02:12:24.938993 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:24.938998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:24.939022 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:24.939164 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:24.939877 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:24.939880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:24.939881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:24.939884 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:24.939901 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:24.939908 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:24.939971 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:24.940037 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 02:12:24.940164 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6189,"thread_start_us":66,"threads_started":1}
I20240501 02:12:24.941362 12346 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:24.941395 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:24.941524 12346 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:24.947752 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:24.942763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49649 (local address 127.22.12.126:44411)
0501 02:12:24.942816 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:24.942818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:24.946742 (+ 3924us) server_negotiation.cc:407] Connection header received
0501 02:12:24.946807 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:24.946808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:24.946814 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:24.946867 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:24.947527 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:24.947528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:24.947528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 02:12:24.947532 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:24.947550 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 02:12:24.947555 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:24.947602 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:24.947657 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 02:12:24.947672 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240501 02:12:24.955770 12346 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f0b26f0d908c4619b55623fa0a19855e: Generated new TSK 0
I20240501 02:12:24.955971 12346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:24.959307 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12266
2024-05-01T02:12:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1141 ms)
[----------] 18 tests from AuthTokenIssuingTest (20934 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:25 dist-test-slave-9tkh krb5kdc[12365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:25 dist-test-slave-9tkh krb5kdc[12365](info): set up 2 sockets
May 01 02:12:25 dist-test-slave-9tkh krb5kdc[12365](info): commencing operation
krb5kdc: starting...
W20240501 02:12:25.850508 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:25 dist-test-slave-9tkh krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:25.959362 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35961
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35961,127.22.12.125:35487,127.22.12.124:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:26.038617 12378 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 02:12:26.038857 12378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:26.038913 12378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:26.038959 12378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:26.040407 12378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:26.040462 12378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:26.040480 12378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:26.040498 12378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:26.040510 12378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:26.042243 12378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35961,127.22.12.125:35487,127.22.12.124:40103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:26.042493 12378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:26.042709 12378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 02:12:26.047755 12378 server_base.cc:1034] running on GCE node
W20240501 02:12:26.048049 12383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:26.048386 12384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:26.048514 12386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:26.048739 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:26.048966 12378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:26.050097 12378 hybrid_clock.cc:648] HybridClock initialized: now 1714529546050081 us; error 28 us; skew 500 ppm
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:26.057888 12378 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:26.058351 12378 webserver.cc:460] Webserver started at http://127.22.12.126:41421/ using document root <none> and password file <none>
I20240501 02:12:26.058560 12378 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:26.058630 12378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:26.058737 12378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:26.060087 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "feb0c7c8768340ea9006f07694f8898b"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "19923fea1ac26ae434f64e3d8979c957"
server_key_iv: "5315a0b4a5deb01a5c82686cb0994609"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.060460 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "feb0c7c8768340ea9006f07694f8898b"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "19923fea1ac26ae434f64e3d8979c957"
server_key_iv: "5315a0b4a5deb01a5c82686cb0994609"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.061995 12378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240501 02:12:26.062870 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:26.063028 12378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:26.063127 12378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "feb0c7c8768340ea9006f07694f8898b"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "19923fea1ac26ae434f64e3d8979c957"
server_key_iv: "5315a0b4a5deb01a5c82686cb0994609"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.063207 12378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:26.084794 12378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:26.087500 12378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:26.087654 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:26.092638 12378 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:35961
I20240501 02:12:26.092669 12445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:35961 every 8 connection(s)
I20240501 02:12:26.093092 12378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:26.093765 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12378
I20240501 02:12:26.093875 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:26.094034 12446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:26.094080 22577 external_mini_cluster.cc:1428] Setting key 33b815c030e840ce1edc6417a353e37d
I20240501 02:12:26.095546 12446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.125@KRBTEST.COM" created.
I20240501 02:12:26.105140 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.096582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.22.12.126:35961)
0501 02:12:26.096720 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.096737 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.096764 (+ 27us) server_negotiation.cc:407] Connection header received
0501 02:12:26.097317 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.097320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.097345 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.097482 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.098408 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.099463 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.100097 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.100323 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.102064 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.102074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.102075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.102081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.103740 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.104065 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.104067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.104067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.104158 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.104312 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.104313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.104314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.104554 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.104652 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.104848 (+ 196us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.104950 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240501 02:12:26.105144 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.096320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35961 (local address 127.0.0.1:57822)
0501 02:12:26.096495 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.096528 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.097148 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.097497 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.097501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.097941 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.098296 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.098303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.099544 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.099544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.100010 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.100012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.100128 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.100651 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.100667 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.101988 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.103820 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.103826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.103828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.103997 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.104210 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.104210 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.104211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.104257 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.104619 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.104623 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.104703 (+ 80us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.104815 (+ 112us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.105019 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":58,"thread_start_us":86,"threads_started":1}
I20240501 02:12:26.105837 12446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:26.106181 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.106055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46746)
0501 02:12:26.106101 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.106154 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240501 02:12:26.106307 12396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
I20240501 02:12:26.106725 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.106635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46752)
0501 02:12:26.106674 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.106704 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240501 02:12:26.106837 12446 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:35487: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab.
Entry for principal kudu/127.22.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab.
I20240501 02:12:26.119808 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:35487
--webserver_interface=127.22.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab
--principal=kudu/127.22.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35961,127.22.12.125:35487,127.22.12.124:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:12:26.156036 12446 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } attempt: 1
I20240501 02:12:26.156512 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.156384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46756)
0501 02:12:26.156448 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.156479 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20240501 02:12:26.156976 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.156882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46766)
0501 02:12:26.156922 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.156948 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240501 02:12:26.157073 12446 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:35487: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
W20240501 02:12:26.199072 12453 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 02:12:26.199301 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:26.199324 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:26.199350 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:26.200734 12453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:26.200786 12453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:26.200804 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:26.200819 12453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:26.200834 12453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:26.202467 12453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35961,127.22.12.125:35487,127.22.12.124:40103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab
--principal=kudu/127.22.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:35487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:26.202749 12453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:26.202937 12453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:26.207136 12461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:26.207486 12459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:26.207965 12458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:26.208398 12453 server_base.cc:1034] running on GCE node
I20240501 02:12:26.208652 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:26.209049 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:26.210222 12453 hybrid_clock.cc:648] HybridClock initialized: now 1714529546210185 us; error 52 us; skew 500 ppm
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:26.212937 12453 init.cc:377] Logged in from keytab as kudu/127.22.12.125@KRBTEST.COM (short username kudu)
I20240501 02:12:26.213363 12453 webserver.cc:460] Webserver started at http://127.22.12.125:40911/ using document root <none> and password file <none>
I20240501 02:12:26.213517 12453 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:26.213552 12453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:26.213640 12453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:26.214727 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data/instance:
uuid: "5227b83d5dc741de84b8781ca6cf7040"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "6633ab3b26813de9c6353c867b7ebbd2"
server_key_iv: "91b55654add9823859d7d45c5ed04cdd"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.215003 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal/instance:
uuid: "5227b83d5dc741de84b8781ca6cf7040"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "6633ab3b26813de9c6353c867b7ebbd2"
server_key_iv: "91b55654add9823859d7d45c5ed04cdd"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.216145 12453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 02:12:26.216861 12468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:26.217069 12453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 02:12:26.217147 12453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal
uuid: "5227b83d5dc741de84b8781ca6cf7040"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "6633ab3b26813de9c6353c867b7ebbd2"
server_key_iv: "91b55654add9823859d7d45c5ed04cdd"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.217201 12453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:26.225230 12446 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } attempt: 2
I20240501 02:12:26.225689 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.225547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46774)
0501 02:12:26.225607 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.225652 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240501 02:12:26.226058 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.225969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46776)
0501 02:12:26.226010 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.226034 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240501 02:12:26.226152 12446 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:35487: Network error: Client connection negotiation failed: client connection to 127.22.12.125:35487: connect: Connection refused (error 111)
I20240501 02:12:26.227277 12453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:26.229358 12453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:26.229480 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:26.233129 12453 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.125:35487
I20240501 02:12:26.233176 12520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.125:35487 every 8 connection(s)
I20240501 02:12:26.233484 12453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
I20240501 02:12:26.234246 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12453
I20240501 02:12:26.234369 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-1/wal/instance
I20240501 02:12:26.234478 12521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:26.234602 22577 external_mini_cluster.cc:1428] Setting key 4c1981110cab17c3ec1f16ac515491f8
I20240501 02:12:26.236371 12521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.12.124@KRBTEST.COM; defaulting to no policy
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.124@KRBTEST.COM" created.
I20240501 02:12:26.244311 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.237102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.22.12.126:35961)
0501 02:12:26.237154 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.237156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.237341 (+ 185us) server_negotiation.cc:407] Connection header received
0501 02:12:26.237850 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.237852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.237863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.237919 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.238688 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.239142 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.239640 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.239814 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.241628 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.241633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.241634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.241641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.242973 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.243341 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.243342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.243342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.243367 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.243559 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.243560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.243561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.243666 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.243731 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.244116 (+ 385us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.244214 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240501 02:12:26.244365 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.237078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35961 (local address 127.0.0.1:57838)
0501 02:12:26.237239 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.237265 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.237740 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.237980 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.237984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.238290 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.238603 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.238607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.239218 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.239219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.239569 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.239571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.239699 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.240238 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.240256 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.241529 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.243059 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.243067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.243070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.243277 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.243417 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.243418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.243419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.243502 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.243753 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.243760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.243907 (+ 147us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.244087 (+ 180us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.244213 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20240501 02:12:26.244926 12521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:12:26.252009 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.245209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46792)
0501 02:12:26.245257 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.245266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.245304 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.245566 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.245566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.245678 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.245822 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.245824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.246994 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.246995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.247365 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.247367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.247434 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.247904 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.247908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.249031 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.251064 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.251065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.251065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.251209 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.251442 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.251444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.251444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.251475 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.251716 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.251717 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.251761 (+ 44us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.251828 (+ 67us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.251933 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20240501 02:12:26.252120 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.245246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.22.12.125:35487)
0501 02:12:26.245414 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.245423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.245440 (+ 17us) server_negotiation.cc:407] Connection header received
0501 02:12:26.245465 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.245466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.245482 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.245561 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.245890 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.246911 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.247449 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.247675 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.249113 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.249125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.249125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.249133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.250973 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.251305 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.251308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.251308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.251346 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.251531 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.251532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.251532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.251645 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.251709 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.251874 (+ 165us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.251997 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20240501 02:12:26.252624 12521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:26.252943 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.252833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56186)
0501 02:12:26.252876 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.252922 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240501 02:12:26.253096 12470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
I20240501 02:12:26.253517 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.253445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56200)
0501 02:12:26.253479 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.253502 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240501 02:12:26.253630 12521 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:40103: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab.
Entry for principal kudu/127.22.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab.
I20240501 02:12:26.259020 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:40103
--webserver_interface=127.22.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab
--principal=kudu/127.22.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35961,127.22.12.125:35487,127.22.12.124:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:12:26.302863 12521 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } attempt: 1
I20240501 02:12:26.303402 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.303252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56212)
0501 02:12:26.303317 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.303366 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 02:12:26.303802 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.303709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56216)
0501 02:12:26.303756 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.303783 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 02:12:26.303917 12521 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:40103: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
I20240501 02:12:26.317328 12446 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } attempt: 3
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:12:26.325044 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.317774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46800)
0501 02:12:26.317847 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.317861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.317938 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.318088 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.318088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.318251 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.318421 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.318423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.319327 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.319329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.319711 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.319713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.319776 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.320328 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.320333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.322195 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.324052 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.324053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.324053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.324205 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.324443 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.324445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.324445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.324481 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.324729 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.324730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.324780 (+ 50us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.324831 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.324965 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20240501 02:12:26.325083 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.317841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.22.12.125:35487)
0501 02:12:26.317910 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.317912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.317928 (+ 16us) server_negotiation.cc:407] Connection header received
0501 02:12:26.317999 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.318000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.318015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.318096 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.318496 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.319228 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.319792 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.320027 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.322295 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.322301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.322302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.322311 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.323930 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.324271 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.324272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.324273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.324303 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.324545 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.324546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.324546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.324666 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.324731 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.324884 (+ 153us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.324979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 02:12:26.325500 12446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:26.325847 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.325742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56220)
0501 02:12:26.325786 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.325814 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 02:12:26.326165 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.326082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56224)
0501 02:12:26.326120 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.326143 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240501 02:12:26.326251 12446 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:40103: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
W20240501 02:12:26.340325 12528 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 02:12:26.340570 12528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:26.340595 12528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:26.340621 12528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:26.342000 12528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:26.342052 12528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:26.342070 12528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:26.342087 12528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:26.342100 12528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:26.343907 12528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35961,127.22.12.125:35487,127.22.12.124:40103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab
--principal=kudu/127.22.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:26.344179 12528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:26.344365 12528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:26.348263 12536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:26.348711 12533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:26.349062 12534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:26.349534 12528 server_base.cc:1034] running on GCE node
I20240501 02:12:26.349771 12528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:26.350021 12528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:26.351161 12528 hybrid_clock.cc:648] HybridClock initialized: now 1714529546351143 us; error 28 us; skew 500 ppm
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:26.353574 12528 init.cc:377] Logged in from keytab as kudu/127.22.12.124@KRBTEST.COM (short username kudu)
I20240501 02:12:26.353979 12528 webserver.cc:460] Webserver started at http://127.22.12.124:46767/ using document root <none> and password file <none>
I20240501 02:12:26.354148 12528 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:26.354188 12528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:26.354290 12528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:26.355304 12528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data/instance:
uuid: "8621b8a9d058426a88d7ab07d56f9e8e"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "1a92b19b9fdea8dc000053ec10a1676a"
server_key_iv: "1448e6d5c3fabe05085219ffe21b22cf"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.355605 12528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal/instance:
uuid: "8621b8a9d058426a88d7ab07d56f9e8e"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "1a92b19b9fdea8dc000053ec10a1676a"
server_key_iv: "1448e6d5c3fabe05085219ffe21b22cf"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.356892 12528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 02:12:26.357378 12446 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } attempt: 1
I20240501 02:12:26.357798 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.357651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56226)
0501 02:12:26.357715 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.357758 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240501 02:12:26.358101 12543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:26.358148 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.358049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56238)
0501 02:12:26.358086 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.358118 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20240501 02:12:26.358408 12528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 02:12:26.358494 12528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal
uuid: "8621b8a9d058426a88d7ab07d56f9e8e"
format_stamp: "Formatted at 2024-05-01 02:12:26 on dist-test-slave-9tkh"
server_key: "1a92b19b9fdea8dc000053ec10a1676a"
server_key_iv: "1448e6d5c3fabe05085219ffe21b22cf"
server_key_version: "encryptionkey@0"
I20240501 02:12:26.358593 12528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 02:12:26.358733 12446 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:40103: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
I20240501 02:12:26.372083 12521 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } attempt: 2
I20240501 02:12:26.372578 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.372446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56242)
0501 02:12:26.372507 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.372549 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20240501 02:12:26.372915 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.372823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56258)
0501 02:12:26.372867 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.372892 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:12:26.373008 12521 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:40103: Network error: Client connection negotiation failed: client connection to 127.22.12.124:40103: connect: Connection refused (error 111)
I20240501 02:12:26.375949 12528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:26.378098 12528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:26.378211 12528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:26.381870 12528 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.124:40103
I20240501 02:12:26.381911 12595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.124:40103 every 8 connection(s)
I20240501 02:12:26.382216 12528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
I20240501 02:12:26.383105 12596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:26.383836 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12528
I20240501 02:12:26.383955 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714529249419714-22577-0/minicluster-data/master-2/wal/instance
I20240501 02:12:26.384099 22577 external_mini_cluster.cc:1428] Setting key 30b89bb1b5f482f62a2a79c63a8b4d40
I20240501 02:12:26.384696 12596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:12:26.392300 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.385347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.22.12.126:35961)
0501 02:12:26.385397 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.385399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.385542 (+ 143us) server_negotiation.cc:407] Connection header received
0501 02:12:26.385604 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.385606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.385617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.385664 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.386362 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.386843 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.387343 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.387508 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.389516 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.389520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.389520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.389528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.391011 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.391465 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.391466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.391466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.391492 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.391676 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.391676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.391677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.391778 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.391841 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.392133 (+ 292us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.392212 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:12:26.392454 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.385297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35961 (local address 127.0.0.1:57842)
0501 02:12:26.385449 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.385471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.385549 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.385689 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.385692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.386012 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.386260 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.386265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.386916 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.386917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.387268 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.387270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.387398 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.387944 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.387967 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.389389 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.391112 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.391117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.391120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.391395 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.391551 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.391552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.391552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.391621 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.391838 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.391843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.392002 (+ 159us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.392101 (+ 99us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.392241 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20240501 02:12:26.392609 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.384610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.22.12.124:40103)
0501 02:12:26.384786 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.384798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.384816 (+ 18us) server_negotiation.cc:407] Connection header received
0501 02:12:26.385318 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.385328 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.385392 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.385538 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.386316 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.387019 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.387539 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.387720 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.389824 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.389834 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.389834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.389841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.391543 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.391890 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.391893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.391893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.391926 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.392113 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.392114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.392114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.392221 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.392357 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.392454 (+ 97us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.392530 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20240501 02:12:26.393390 12596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:26.399500 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.393684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.22.12.125:35487)
0501 02:12:26.393726 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.393728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.393761 (+ 33us) server_negotiation.cc:407] Connection header received
0501 02:12:26.393820 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.393821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.393831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.393900 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.394247 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.394790 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.395289 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.395447 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.396981 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.396986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.396986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.396993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.398577 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.398852 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.398853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.398854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.398879 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.399012 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.399013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.399014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.399133 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.399189 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.399318 (+ 129us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.399398 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240501 02:12:26.399502 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.393617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:35487 (local address 127.0.0.1:46804)
0501 02:12:26.393724 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.393733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.393772 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.393903 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.393903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.394017 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.394175 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.394177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.394852 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.394853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.395218 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.395219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.395275 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.395764 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.395768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.396893 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.398652 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.398653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.398654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.398788 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.398932 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.398933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.398933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.398962 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.399184 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.399185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.399231 (+ 46us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.399289 (+ 58us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.399402 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20240501 02:12:26.399765 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.393644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.22.12.126:35961)
0501 02:12:26.393677 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.393678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.393697 (+ 19us) server_negotiation.cc:407] Connection header received
0501 02:12:26.393716 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.393717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.393726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.393775 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.394382 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.394830 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.395299 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.395452 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.397478 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.397482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.397483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.397489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.398836 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.399128 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.399128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.399129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.399145 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.399297 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.399298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.399298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.399381 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.399509 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.399563 (+ 54us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.399667 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240501 02:12:26.400398 12596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:12:26.406666 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.400640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56268)
0501 02:12:26.400747 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.400755 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.400796 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.400939 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.400939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.401062 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.401298 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.401301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.401889 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.401890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.402224 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.402225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.402277 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.402739 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.402743 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.403946 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.405795 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.405796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.405796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.405920 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.406067 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.406067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.406067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.406102 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.406314 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.406314 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.406347 (+ 33us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.406408 (+ 61us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.406502 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20240501 02:12:26.406715 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.400680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.22.12.124:40103)
0501 02:12:26.400722 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.400724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.400783 (+ 59us) server_negotiation.cc:407] Connection header received
0501 02:12:26.400879 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.400880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.400887 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.400928 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.401379 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.401827 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.402291 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.402443 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.404035 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.404039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.404039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.404046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.405595 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.405985 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.405986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.405986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.406010 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.406149 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.406150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.406150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.406262 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.406326 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.406438 (+ 112us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.406581 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":23}
I20240501 02:12:26.407537 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.401181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.22.12.125:35487)
0501 02:12:26.401213 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.401215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.401222 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:26.401239 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.401240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.401247 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.401303 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.401790 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.402229 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.402898 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.403027 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.405041 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.405045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.405045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.405052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.406316 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.406653 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.406654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.406655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.406679 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.406922 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.406924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.406924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.407013 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.407158 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.407191 (+ 33us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.407445 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:12:26.408545 12596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: Bootstrap starting.
I20240501 02:12:26.409008 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:26.409180 12596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:26.409431 12596 log.cc:824] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:26.410182 12596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: No bootstrap required, opened a new log
I20240501 02:12:26.409706 12420 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "feb0c7c8768340ea9006f07694f8898b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57852
I20240501 02:12:26.411499 12596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.411613 12596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8621b8a9d058426a88d7ab07d56f9e8e, State: Initialized, Role: FOLLOWER
I20240501 02:12:26.411676 12596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.411916 12602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.411967 12596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:26.412014 12602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: This master's current role is: FOLLOWER
W20240501 02:12:26.413556 12613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:26.413652 12613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:26.414036 12528 master_runner.cc:186] Error getting master registration for 127.22.12.126:35961: OK, instance_id { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" instance_seqno: 1714529546091204 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 02:12:26.416465 12528 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 02:12:26.433918 12446 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } attempt: 2
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:12:26.440934 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.434401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.22.12.124:40103)
0501 02:12:26.434442 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.434444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.434453 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:12:26.434548 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.434549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.434560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.434610 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.435028 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.435452 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.435975 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.436138 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.438137 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.438142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.438143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.438150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.439964 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.440265 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.440266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.440267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.440292 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.440412 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.440412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.440413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.440543 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.440598 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.440741 (+ 143us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.440821 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240501 02:12:26.440928 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.434299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56270)
0501 02:12:26.434369 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.434383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.434470 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.434629 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.434630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.434799 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.434937 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.434938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.435527 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.435528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.435868 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.435869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.435933 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.436507 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.436512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.438041 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.440054 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.440055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.440056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.440203 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.440335 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.440335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.440336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.440365 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.440600 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.440601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.440650 (+ 49us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.440711 (+ 61us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.440825 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240501 02:12:26.442627 12446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b: Bootstrap starting.
I20240501 02:12:26.443328 12446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:26.443636 12446 log.cc:824] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:26.444468 12446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b: No bootstrap required, opened a new log
I20240501 02:12:26.445710 12446 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.445852 12446 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb0c7c8768340ea9006f07694f8898b, State: Initialized, Role: FOLLOWER
I20240501 02:12:26.445911 12446 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.446235 12446 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:26.446198 12616 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.446298 12616 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:26.447736 12420 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 02:12:26.447820 12420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:26.448967 12420 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.449252 12420 leader_election.cc:288] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [CANDIDATE]: Term 1 election: Requested vote from peers 5227b83d5dc741de84b8781ca6cf7040 (127.22.12.125:35487), 8621b8a9d058426a88d7ab07d56f9e8e (127.22.12.124:40103)
W20240501 02:12:26.449832 12627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:26.449921 12627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:26.450714 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "feb0c7c8768340ea9006f07694f8898b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8621b8a9d058426a88d7ab07d56f9e8e"
I20240501 02:12:26.450845 12570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:26.451745 12570 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb0c7c8768340ea9006f07694f8898b in term 1.
I20240501 02:12:26.451979 12395 leader_election.cc:302] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8621b8a9d058426a88d7ab07d56f9e8e, feb0c7c8768340ea9006f07694f8898b; no voters:
I20240501 02:12:26.452112 12616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:26.452237 12616 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [term 1 LEADER]: Becoming Leader. State: Replica: feb0c7c8768340ea9006f07694f8898b, State: Running, Role: LEADER
I20240501 02:12:26.452330 12616 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.452708 12616 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb0c7c8768340ea9006f07694f8898b. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.452780 12616 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:26.454128 12378 master_runner.cc:186] Error getting master registration for 127.22.12.125:35487: OK, instance_id { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" instance_seqno: 1714529546232114 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 02:12:26.456148 12378 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 02:12:26.460880 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.451060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.22.12.126:35961)
0501 02:12:26.451106 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.451107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.451650 (+ 543us) server_negotiation.cc:407] Connection header received
0501 02:12:26.451794 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.451795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.451807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.451836 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.453605 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.454244 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.455266 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.455440 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.457498 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.457502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.457503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.457522 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.459375 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.459901 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.459902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.459902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.459940 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.460170 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.460170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.460171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.460294 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.460377 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.460690 (+ 313us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.460773 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:26.461763 12638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:26.462038 12638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:26.462838 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.450999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.22.12.124:40103)
0501 02:12:26.451044 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.451046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.452428 (+ 1382us) server_negotiation.cc:407] Connection header received
0501 02:12:26.452665 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.452666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.452677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.452726 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.454836 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.455464 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.458683 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.458874 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.460001 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.460006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.460006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.460013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.461659 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.462002 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.462003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.462004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.462027 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.462245 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.462246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.462247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.462354 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.462420 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.462659 (+ 239us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.462747 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:12:26.464157 12521 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } attempt: 3
May 01 02:12:26 dist-test-slave-9tkh krb5kdc[12365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529546, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:12:26.469254 12570 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [term 1 FOLLOWER]: Refusing update from remote peer feb0c7c8768340ea9006f07694f8898b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:26.469678 12636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:26.471031 12602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb0c7c8768340ea9006f07694f8898b. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.471132 12602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:26.472093 12639 mvcc.cc:205] Tried to move back new op lower bound from 7022713022335066112 to 7022713022269345792. Current Snapshot: MvccSnapshot[applied={T|T < 7022713022335066112}]
I20240501 02:12:26.472188 12636 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.472265 12636 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:26.472470 12602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.472559 12602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:26.473134 12640 mvcc.cc:205] Tried to move back new op lower bound from 7022713022335066112 to 7022713022269345792. Current Snapshot: MvccSnapshot[applied={T|T < 7022713022335066112}]
I20240501 02:12:26.473832 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.464445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:40103 (local address 127.0.0.1:56298)
0501 02:12:26.464496 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:26.464506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:26.464538 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:26.464716 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:26.464717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:26.464894 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:26.465074 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.465076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.465964 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.465965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:26.466508 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:26.466510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.466614 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.467098 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:26.467103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:26.468519 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:26.472645 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.472646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.472647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.472840 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.473097 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:26.473097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:26.473098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:26.473144 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:26.473535 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:26.473536 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:26.473584 (+ 48us) client_negotiation.cc:769] Sending connection context
0501 02:12:26.473628 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 02:12:26.473734 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20240501 02:12:26.474421 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.464506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.22.12.124:40103)
0501 02:12:26.464568 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.464570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.464584 (+ 14us) server_negotiation.cc:407] Connection header received
0501 02:12:26.464602 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.464603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.464613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.464667 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.465158 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.465861 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.466783 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.466952 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.468613 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.468616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.468617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.468623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.472489 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.472950 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.472951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.472951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.473006 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.473222 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.473223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.473223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.473423 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.474039 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.474109 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.474278 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240501 02:12:26.475909 12521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: Bootstrap starting.
I20240501 02:12:26.476243 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:26.451307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.22.12.125:35487)
0501 02:12:26.451341 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.451343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.461803 (+ 10460us) server_negotiation.cc:407] Connection header received
0501 02:12:26.461951 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.461952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.461965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.462004 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.463726 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.464403 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:26.465526 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:26.465727 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:26.468502 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:26.468505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:26.468506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:26.468512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:26.470465 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.475266 (+ 4801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.475267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.475268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.475294 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:26.475517 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:26.475518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:26.475518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:26.475644 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:26.475735 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:26.476070 (+ 335us) server_negotiation.cc:299] Negotiation successful
0501 02:12:26.476156 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:26.476569 12521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:26.476845 12521 log.cc:824] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:26.478924 12638 catalog_manager.cc:1347] Generated new cluster ID: 9697525f31e14c0b89779cda24f547aa
I20240501 02:12:26.478981 12638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 02:12:26.483513 12521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: No bootstrap required, opened a new log
I20240501 02:12:26.484190 12638 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 02:12:26.485018 12638 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:26.485386 12521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.485532 12521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5227b83d5dc741de84b8781ca6cf7040, State: Initialized, Role: FOLLOWER
I20240501 02:12:26.485605 12521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } }
I20240501 02:12:26.485935 12521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:26.487440 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "feb0c7c8768340ea9006f07694f8898b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5227b83d5dc741de84b8781ca6cf7040"
I20240501 02:12:26.487555 12495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:26.488468 12495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb0c7c8768340ea9006f07694f8898b in term 1.
I20240501 02:12:26.488886 12494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [term 1 FOLLOWER]: Refusing update from remote peer feb0c7c8768340ea9006f07694f8898b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:26.489151 12616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:26.491024 12638 catalog_manager.cc:5857] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b: Generated new TSK 0
I20240501 02:12:26.491230 12638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 02:12:26.491672 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.491772 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:26.491955 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb0c7c8768340ea9006f07694f8898b. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.492000 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:26.492026 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.492105 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:26.495061 12649 mvcc.cc:205] Tried to move back new op lower bound from 7022713022393159680 to 7022713022269345792. Current Snapshot: MvccSnapshot[applied={T|T < 7022713022335066112}]
I20240501 02:12:26.495831 12636 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "feb0c7c8768340ea9006f07694f8898b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb0c7c8768340ea9006f07694f8898b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35961 } } peers { permanent_uuid: "5227b83d5dc741de84b8781ca6cf7040" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 35487 } } peers { permanent_uuid: "8621b8a9d058426a88d7ab07d56f9e8e" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 40103 } } }
I20240501 02:12:26.495934 12636 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb0c7c8768340ea9006f07694f8898b [sys.catalog]: This master's current role is: LEADER
W20240501 02:12:26.496232 12661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:26.496289 12661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 02:12:26.514171 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.512733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.22.12.125:35487)
0501 02:12:26.512776 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.512778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.513441 (+ 663us) server_negotiation.cc:407] Connection header received
0501 02:12:26.513602 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.513603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.513615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.513643 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.514100 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46826: BlockingRecv error: recv got EOF from 127.0.0.1:46826 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240501 02:12:26.514950 12599 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.513140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.22.12.124:40103)
0501 02:12:26.513180 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.513182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.514426 (+ 1244us) server_negotiation.cc:407] Connection header received
0501 02:12:26.514518 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.514519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.514555 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.514585 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.514892 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56314: BlockingRecv error: recv got EOF from 127.0.0.1:56314 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240501 02:12:26.515969 12450 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.515344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.22.12.126:35961)
0501 02:12:26.515396 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.515398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.515490 (+ 92us) server_negotiation.cc:407] Connection header received
0501 02:12:26.515571 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.515572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.515582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.515608 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.515916 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57858: BlockingRecv error: recv got EOF from 127.0.0.1:57858 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20240501 02:12:26.518294 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.517533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.22.12.125:35487)
0501 02:12:26.517570 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.517572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.517773 (+ 201us) server_negotiation.cc:407] Connection header received
0501 02:12:26.517878 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.517879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.517895 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.517921 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.518246 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46830: BlockingRecv error: recv got EOF from 127.0.0.1:46830 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20240501 02:12:26.521524 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.520688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.22.12.125:35487)
0501 02:12:26.520722 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.520723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.520978 (+ 255us) server_negotiation.cc:407] Connection header received
0501 02:12:26.521091 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.521092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.521103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.521133 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.521471 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46840: BlockingRecv error: recv got EOF from 127.0.0.1:46840 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20240501 02:12:26.525549 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.524705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.22.12.125:35487)
0501 02:12:26.524743 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.524744 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.525015 (+ 271us) server_negotiation.cc:407] Connection header received
0501 02:12:26.525126 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.525127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.525138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.525164 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.525496 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46854: BlockingRecv error: recv got EOF from 127.0.0.1:46854 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20240501 02:12:26.531785 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.530848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.22.12.125:35487)
0501 02:12:26.530886 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.530888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.531192 (+ 304us) server_negotiation.cc:407] Connection header received
0501 02:12:26.531300 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.531302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.531314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.531343 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.531726 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46856: BlockingRecv error: recv got EOF from 127.0.0.1:46856 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240501 02:12:26.542174 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.541165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.22.12.125:35487)
0501 02:12:26.541210 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.541212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.541532 (+ 320us) server_negotiation.cc:407] Connection header received
0501 02:12:26.541643 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.541645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.541658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.541689 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.542111 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46870: BlockingRecv error: recv got EOF from 127.0.0.1:46870 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20240501 02:12:26.560611 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.559611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.22.12.125:35487)
0501 02:12:26.559656 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.559659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.559974 (+ 315us) server_negotiation.cc:407] Connection header received
0501 02:12:26.560090 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.560091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.560103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.560132 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.560528 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46880: BlockingRecv error: recv got EOF from 127.0.0.1:46880 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240501 02:12:26.595021 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.594039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.22.12.125:35487)
0501 02:12:26.594095 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.594098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.594408 (+ 310us) server_negotiation.cc:407] Connection header received
0501 02:12:26.594504 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.594505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.594516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.594579 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.594962 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46888: BlockingRecv error: recv got EOF from 127.0.0.1:46888 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240501 02:12:26.661154 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.660436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.22.12.125:35487)
0501 02:12:26.660482 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.660484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.660586 (+ 102us) server_negotiation.cc:407] Connection header received
0501 02:12:26.660640 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.660641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.660653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.660733 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.661092 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46898: BlockingRecv error: recv got EOF from 127.0.0.1:46898 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240501 02:12:26.791041 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:26.790323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.22.12.125:35487)
0501 02:12:26.790372 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:26.790373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:26.790487 (+ 114us) server_negotiation.cc:407] Connection header received
0501 02:12:26.790581 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:26.790581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:26.790595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:26.790643 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:26.790973 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46906: BlockingRecv error: recv got EOF from 127.0.0.1:46906 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20240501 02:12:27.048761 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:27.048205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.22.12.125:35487)
0501 02:12:27.048253 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:27.048255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:27.048283 (+ 28us) server_negotiation.cc:407] Connection header received
0501 02:12:27.048346 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:27.048347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:27.048357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:27.048405 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:27.048700 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46914: BlockingRecv error: recv got EOF from 127.0.0.1:46914 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:12:27.414043 12613 catalog_manager.cc:1259] Loaded cluster ID: 9697525f31e14c0b89779cda24f547aa
I20240501 02:12:27.414125 12613 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: loading cluster ID for follower catalog manager: success
I20240501 02:12:27.415028 12613 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: acquiring CA information for follower catalog manager: success
I20240501 02:12:27.415315 12613 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8621b8a9d058426a88d7ab07d56f9e8e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:12:27.496695 12661 catalog_manager.cc:1259] Loaded cluster ID: 9697525f31e14c0b89779cda24f547aa
I20240501 02:12:27.496783 12661 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: loading cluster ID for follower catalog manager: success
I20240501 02:12:27.497720 12661 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: acquiring CA information for follower catalog manager: success
I20240501 02:12:27.498044 12661 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5227b83d5dc741de84b8781ca6cf7040: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:12:27.563946 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:27.561908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.22.12.125:35487)
0501 02:12:27.562025 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:27.562027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:27.562040 (+ 13us) server_negotiation.cc:407] Connection header received
0501 02:12:27.562105 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:27.562106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:27.562145 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:27.562226 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:27.562472 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.562984 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:27.563520 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.563627 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:27.563664 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:27.563752 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:27.563809 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:27.563841 (+ 32us) server_negotiation.cc:299] Negotiation successful
0501 02:12:27.563861 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20240501 02:12:27.567670 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:27.565645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.22.12.124:40103)
0501 02:12:27.565774 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:27.565776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:27.565787 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:12:27.565872 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:27.565872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:27.565897 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:27.565955 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:27.566215 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.566734 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:27.567240 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.567350 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:27.567384 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:27.567469 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:27.567527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:27.567572 (+ 45us) server_negotiation.cc:299] Negotiation successful
0501 02:12:27.567593 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20240501 02:12:27.571537 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:27.569301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.22.12.124:40103)
0501 02:12:27.569378 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:27.569380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:27.569388 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:27.569408 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:27.569408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:27.569424 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:27.569486 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:27.569760 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.570465 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:27.571184 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.571304 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:27.571317 (+ 13us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:27.571337 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:27.571380 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:27.571446 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 02:12:27.571466 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 02:12:27.571648 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:27.569172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.22.12.125:35487)
0501 02:12:27.569212 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:27.569214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:27.569436 (+ 222us) server_negotiation.cc:407] Connection header received
0501 02:12:27.569493 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:27.569494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:27.569508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:27.569558 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:27.569853 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.570577 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:27.571132 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.571300 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:27.571402 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:27.571428 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:27.571451 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:27.571552 (+ 101us) server_negotiation.cc:299] Negotiation successful
0501 02:12:27.571577 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240501 02:12:27.571846 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:27.569283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.22.12.126:35961)
0501 02:12:27.569405 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:27.569407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:27.569414 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:27.569465 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:27.569465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:27.569496 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:27.569566 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:27.569879 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.570354 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:27.571109 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:27.571302 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:27.571363 (+ 61us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:27.571460 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:27.571528 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:27.571688 (+ 160us) server_negotiation.cc:299] Negotiation successful
0501 02:12:27.571777 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20240501 02:12:27.573223 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12378
I20240501 02:12:27.578012 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12453
I20240501 02:12:27.582484 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12528
2024-05-01T02:12:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2624 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:12:27 dist-test-slave-9tkh krb5kdc[12746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:12:27 dist-test-slave-9tkh krb5kdc[12746](info): set up 2 sockets
May 01 02:12:27 dist-test-slave-9tkh krb5kdc[12746](info): commencing operation
krb5kdc: starting...
W20240501 02:12:28.478691 22577 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/test-user.keytab.
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T02:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T02:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
Entry for principal kudu/127.22.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab.
I20240501 02:12:28.590196 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35711
--webserver_interface=127.22.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35711,127.22.12.125:41143,127.22.12.124:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 02:12:28.668383 12759 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 02:12:28.668588 12759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:28.668603 12759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:28.668622 12759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:28.669950 12759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:28.669992 12759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:28.670004 12759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:28.670014 12759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:28.670023 12759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:28.671537 12759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35711,127.22.12.125:41143,127.22.12.124:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.126.keytab
--principal=kudu/127.22.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.126:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:28.671748 12759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:28.671941 12759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:28.675997 12767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.676378 12765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.676759 12764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:28.677196 12759 server_base.cc:1034] running on GCE node
I20240501 02:12:28.677436 12759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:28.677656 12759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:28.678792 12759 hybrid_clock.cc:648] HybridClock initialized: now 1714529548678780 us; error 31 us; skew 500 ppm
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:28.681314 12759 init.cc:377] Logged in from keytab as kudu/127.22.12.126@KRBTEST.COM (short username kudu)
I20240501 02:12:28.681751 12759 webserver.cc:460] Webserver started at http://127.22.12.126:37027/ using document root <none> and password file <none>
I20240501 02:12:28.681958 12759 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:28.682001 12759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:28.682104 12759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:28.683158 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data/instance:
uuid: "aa564e16990d4fc0a046b7cda32f222b"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "43d6bb49d4c9f1e1c1b72958e946c8b0"
server_key_iv: "14a7123059077d9bd30caa845bf17b3b"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.683457 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal/instance:
uuid: "aa564e16990d4fc0a046b7cda32f222b"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "43d6bb49d4c9f1e1c1b72958e946c8b0"
server_key_iv: "14a7123059077d9bd30caa845bf17b3b"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.684669 12759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:28.685355 12774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:28.685509 12759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:28.685587 12759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal
uuid: "aa564e16990d4fc0a046b7cda32f222b"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "43d6bb49d4c9f1e1c1b72958e946c8b0"
server_key_iv: "14a7123059077d9bd30caa845bf17b3b"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.685647 12759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:28.707166 12759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:28.708978 12759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:28.709100 12759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:28.712889 12759 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.126:35711
I20240501 02:12:28.712924 12826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.126:35711 every 8 connection(s)
I20240501 02:12:28.713228 12759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/data/info.pb
I20240501 02:12:28.714093 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:28.714565 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12759
I20240501 02:12:28.714658 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-0/wal/instance
I20240501 02:12:28.714810 22577 external_mini_cluster.cc:1428] Setting key 69fc9163fee3dbcbeb9d0372c36ce29a
I20240501 02:12:28.715672 12827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.125@KRBTEST.COM" created.
I20240501 02:12:28.724799 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.716512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.22.12.126:35711)
0501 02:12:28.716618 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:28.716627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:28.716728 (+ 101us) server_negotiation.cc:407] Connection header received
0501 02:12:28.717269 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:28.717272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:28.717292 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:28.717384 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:28.718063 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.718856 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.719584 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.719718 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.721931 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:28.721941 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:28.721941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:28.721947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:28.723533 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.723869 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.723871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.723871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.723910 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.724104 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.724105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.724105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.724293 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:28.724351 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:28.724533 (+ 182us) server_negotiation.cc:299] Negotiation successful
0501 02:12:28.724640 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":40,"threads_started":1}
I20240501 02:12:28.724900 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.716340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35711 (local address 127.0.0.1:40300)
0501 02:12:28.716604 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.716637 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:28.717120 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:28.717398 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:28.717400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:28.717720 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:28.717969 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.717974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.718944 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.718945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:28.719508 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.719510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.719661 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.720209 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:28.720233 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:28.721826 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:28.723603 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.723608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.723611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.723798 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.723987 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.723988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.723989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.724046 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.724344 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:28.724347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:28.724414 (+ 67us) client_negotiation.cc:769] Sending connection context
0501 02:12:28.724519 (+ 105us) client_negotiation.cc:240] Negotiation successful
0501 02:12:28.724744 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":87,"thread_start_us":90,"threads_started":1}
I20240501 02:12:28.725605 12827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:28.725929 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.725825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50870)
0501 02:12:28.725869 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.725910 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240501 02:12:28.726038 12778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
I20240501 02:12:28.726462 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.726350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50874)
0501 02:12:28.726405 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.726441 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20240501 02:12:28.726595 12827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:41143: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab.
Entry for principal kudu/127.22.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab.
I20240501 02:12:28.738009 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:41143
--webserver_interface=127.22.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab
--principal=kudu/127.22.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35711,127.22.12.125:41143,127.22.12.124:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:12:28.775826 12827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } attempt: 1
I20240501 02:12:28.776383 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.776230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50884)
0501 02:12:28.776303 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.776348 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20240501 02:12:28.776751 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.776657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50886)
0501 02:12:28.776699 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.776731 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240501 02:12:28.776842 12827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:41143: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
W20240501 02:12:28.815702 12834 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 02:12:28.815909 12834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:28.815924 12834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:28.815943 12834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:28.817265 12834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:28.817310 12834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:28.817322 12834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:28.817333 12834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:28.817343 12834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:28.818899 12834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35711,127.22.12.125:41143,127.22.12.124:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.125.keytab
--principal=kudu/127.22.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.125:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:28.819108 12834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:28.819267 12834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:28.823164 12842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.823480 12839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.823936 12840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:28.824362 12834 server_base.cc:1034] running on GCE node
I20240501 02:12:28.824594 12834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:28.824831 12834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:28.825974 12834 hybrid_clock.cc:648] HybridClock initialized: now 1714529548825958 us; error 35 us; skew 500 ppm
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:28.833094 12834 init.cc:377] Logged in from keytab as kudu/127.22.12.125@KRBTEST.COM (short username kudu)
I20240501 02:12:28.833532 12834 webserver.cc:460] Webserver started at http://127.22.12.125:45351/ using document root <none> and password file <none>
I20240501 02:12:28.833711 12834 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:28.833752 12834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:28.833859 12834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:28.834903 12834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data/instance:
uuid: "2768d71ddca74b8fa53aac3f8b1c7450"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "7a68aec37fafa88a44e301f56e20729c"
server_key_iv: "ae56ee2aebd57ac79ffdddc59631cea6"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.835206 12834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal/instance:
uuid: "2768d71ddca74b8fa53aac3f8b1c7450"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "7a68aec37fafa88a44e301f56e20729c"
server_key_iv: "ae56ee2aebd57ac79ffdddc59631cea6"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.836347 12834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:28.836987 12849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:28.837132 12834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:28.837208 12834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal
uuid: "2768d71ddca74b8fa53aac3f8b1c7450"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "7a68aec37fafa88a44e301f56e20729c"
server_key_iv: "ae56ee2aebd57ac79ffdddc59631cea6"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.837266 12834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:28.845011 12827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } attempt: 2
I20240501 02:12:28.845463 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.845312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50896)
0501 02:12:28.845376 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.845422 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240501 02:12:28.845804 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.845732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50898)
0501 02:12:28.845764 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.845787 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20240501 02:12:28.845888 12827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.125:41143: Network error: Client connection negotiation failed: client connection to 127.22.12.125:41143: connect: Connection refused (error 111)
I20240501 02:12:28.849334 12834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:28.851104 12834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:28.851210 12834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:28.854934 12834 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.125:41143
I20240501 02:12:28.854979 12901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.125:41143 every 8 connection(s)
I20240501 02:12:28.855288 12834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/data/info.pb
I20240501 02:12:28.856106 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:28.857508 12902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:28.862301 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12834
I20240501 02:12:28.862393 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-1/wal/instance
I20240501 02:12:28.862567 22577 external_mini_cluster.cc:1428] Setting key 504284e9558582a06ec92bdf440a58b6
I20240501 02:12:28.865371 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.858166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.22.12.126:35711)
0501 02:12:28.858234 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:28.858237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:28.858357 (+ 120us) server_negotiation.cc:407] Connection header received
0501 02:12:28.858868 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:28.858869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:28.858878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:28.858934 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:28.859657 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.860103 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.860613 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.860740 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.862606 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:28.862612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:28.862613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:28.862620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:28.864058 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.864568 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.864569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.864570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.864591 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.864799 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.864799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.864800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.864923 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:28.865002 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:28.865197 (+ 195us) server_negotiation.cc:299] Negotiation successful
0501 02:12:28.865282 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:12:28.865419 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.858104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35711 (local address 127.0.0.1:40314)
0501 02:12:28.858254 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.858282 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:28.858779 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:28.858989 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:28.858992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:28.859285 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:28.859576 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.859581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.860171 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.860172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:28.860534 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.860535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.860642 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.861130 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:28.861144 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:28.862485 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:28.864162 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.864169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.864173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.864480 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.864655 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.864656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.864656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.864743 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.864978 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:28.864981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:28.865085 (+ 104us) client_negotiation.cc:769] Sending connection context
0501 02:12:28.865166 (+ 81us) client_negotiation.cc:240] Negotiation successful
0501 02:12:28.865276 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20240501 02:12:28.866214 12902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.12.124@KRBTEST.COM; defaulting to no policy
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.12.124@KRBTEST.COM" created.
I20240501 02:12:28.873095 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.866455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50912)
0501 02:12:28.866509 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.866521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:28.866623 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:28.866787 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:28.866788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:28.866934 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:28.867135 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.867137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.867923 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.867925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:28.868460 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.868463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.868541 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.868955 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:28.868960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:28.870126 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:28.872091 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.872092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.872092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.872341 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.872512 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.872512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.872513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.872540 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.872745 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:28.872746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:28.872781 (+ 35us) client_negotiation.cc:769] Sending connection context
0501 02:12:28.872846 (+ 65us) client_negotiation.cc:240] Negotiation successful
0501 02:12:28.872975 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20240501 02:12:28.873260 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.866491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.22.12.125:41143)
0501 02:12:28.866662 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:28.866670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:28.866682 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:12:28.866705 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:28.866707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:28.866717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:28.866771 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:28.867213 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.867853 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.868537 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.868688 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.870210 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:28.870219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:28.870219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:28.870224 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:28.872015 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.872426 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.872429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.872429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.872458 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.872588 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.872589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.872589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.872696 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:28.872769 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:28.872877 (+ 108us) server_negotiation.cc:299] Negotiation successful
0501 02:12:28.873053 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20240501 02:12:28.873805 12902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:28.874151 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.874034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36596)
0501 02:12:28.874077 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.874125 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240501 02:12:28.874276 12851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
I20240501 02:12:28.874691 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.874622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36600)
0501 02:12:28.874651 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.874673 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20240501 02:12:28.874804 12902 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:42963: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab.
Entry for principal kudu/127.22.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab.
I20240501 02:12:28.885454 22577 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKBEtG/build/release/bin/kudu
/tmp/dist-test-taskBKBEtG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:42963
--webserver_interface=127.22.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab
--principal=kudu/127.22.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.12.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.12.126:35711,127.22.12.125:41143,127.22.12.124:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 02:12:28.924015 12902 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } attempt: 1
I20240501 02:12:28.924551 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.924411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36608)
0501 02:12:28.924481 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.924522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 02:12:28.924916 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.924806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36614)
0501 02:12:28.924854 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.924889 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 02:12:28.925026 12902 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:42963: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
I20240501 02:12:28.937053 12827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } attempt: 3
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:12:28.943873 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.937410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50918)
0501 02:12:28.937472 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.937483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:28.937525 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:28.937672 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:28.937673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:28.937843 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:28.938045 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.938047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.938811 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.938813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:28.939189 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:28.939191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.939283 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.940268 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:28.940273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:28.941525 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:28.943040 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.943041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.943042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.943176 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.943307 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:28.943308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:28.943308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:28.943338 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:28.943550 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:28.943551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:28.943597 (+ 46us) client_negotiation.cc:769] Sending connection context
0501 02:12:28.943654 (+ 57us) client_negotiation.cc:240] Negotiation successful
0501 02:12:28.943769 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20240501 02:12:28.943869 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.937502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.22.12.125:41143)
0501 02:12:28.937567 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:28.937569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:28.937579 (+ 10us) server_negotiation.cc:407] Connection header received
0501 02:12:28.937603 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:28.937604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:28.937616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:28.937660 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:28.938214 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.938735 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:28.939280 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:28.939456 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:28.941613 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:28.941617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:28.941617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:28.941624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:28.942961 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.943236 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.943237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.943238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.943258 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:28.943388 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:28.943389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:28.943390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:28.943496 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:28.943550 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:28.943681 (+ 131us) server_negotiation.cc:299] Negotiation successful
0501 02:12:28.943766 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 02:12:28.944272 12827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:28.944592 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.944497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36624)
0501 02:12:28.944540 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.944566 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240501 02:12:28.944939 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.944861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36630)
0501 02:12:28.944901 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.944921 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20240501 02:12:28.945022 12827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:42963: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
W20240501 02:12:28.965009 12909 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 02:12:28.965205 12909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 02:12:28.965219 12909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 02:12:28.965236 12909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 02:12:28.966490 12909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 02:12:28.966549 12909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 02:12:28.966569 12909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 02:12:28.966585 12909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 02:12:28.966614 12909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 02:12:28.968091 12909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.12.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.12.126:35711,127.22.12.125:41143,127.22.12.124:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/krb5kdc/kudu_127.22.12.124.keytab
--principal=kudu/127.22.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.12.124:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type RELEASE
built by None at 01 May 2024 01:43:15 UTC on efd1337a2aa3
build id 66
I20240501 02:12:28.968291 12909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 02:12:28.968466 12909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 02:12:28.972638 12915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.972869 12917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 02:12:28.973062 12914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 02:12:28.973476 12909 server_base.cc:1034] running on GCE node
I20240501 02:12:28.973701 12909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 02:12:28.973924 12909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 02:12:28.975064 12909 hybrid_clock.cc:648] HybridClock initialized: now 1714529548975046 us; error 30 us; skew 500 ppm
I20240501 02:12:28.976173 12827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } attempt: 1
I20240501 02:12:28.976524 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.976418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36644)
0501 02:12:28.976471 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.976494 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240501 02:12:28.976840 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.976762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36660)
0501 02:12:28.976800 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.976821 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240501 02:12:28.976940 12827 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:42963: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
May 01 02:12:28 dist-test-slave-9tkh krb5kdc[12746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 02:12:28.977469 12909 init.cc:377] Logged in from keytab as kudu/127.22.12.124@KRBTEST.COM (short username kudu)
I20240501 02:12:28.977855 12909 webserver.cc:460] Webserver started at http://127.22.12.124:45965/ using document root <none> and password file <none>
I20240501 02:12:28.978029 12909 fs_manager.cc:359] Metadata directory not provided
I20240501 02:12:28.978082 12909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 02:12:28.978192 12909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 02:12:28.979179 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data/instance:
uuid: "15dd698716d24b9ebc46afbff3d7d8f9"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "e1068219b3800b6d020ec0eaff5c41ff"
server_key_iv: "ef0df69aa94f63ad32b8237a7b1edfe9"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.979437 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal/instance:
uuid: "15dd698716d24b9ebc46afbff3d7d8f9"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "e1068219b3800b6d020ec0eaff5c41ff"
server_key_iv: "ef0df69aa94f63ad32b8237a7b1edfe9"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.980492 12909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 02:12:28.981082 12924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 02:12:28.981238 12909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 02:12:28.981313 12909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal
uuid: "15dd698716d24b9ebc46afbff3d7d8f9"
format_stamp: "Formatted at 2024-05-01 02:12:28 on dist-test-slave-9tkh"
server_key: "e1068219b3800b6d020ec0eaff5c41ff"
server_key_iv: "ef0df69aa94f63ad32b8237a7b1edfe9"
server_key_version: "encryptionkey@0"
I20240501 02:12:28.981371 12909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 02:12:28.993168 12902 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } attempt: 2
I20240501 02:12:28.993664 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.993523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36674)
0501 02:12:28.993579 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.993625 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240501 02:12:28.993990 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:28.993898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36686)
0501 02:12:28.993941 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:28.993965 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240501 02:12:28.994079 12902 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.12.124:42963: Network error: Client connection negotiation failed: client connection to 127.22.12.124:42963: connect: Connection refused (error 111)
I20240501 02:12:28.995221 12909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 02:12:28.996851 12909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 02:12:28.996963 12909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 02:12:29.000494 12909 rpc_server.cc:307] RPC server started. Bound to: 127.22.12.124:42963
I20240501 02:12:29.000522 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.12.124:42963 every 8 connection(s)
I20240501 02:12:29.000813 12909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/data/info.pb
I20240501 02:12:29.001665 12977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 02:12:29.003077 12977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:29.009939 22577 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKBEtG/build/release/bin/kudu as pid 12909
I20240501 02:12:29.010048 22577 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKBEtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714529249419714-22577-0/minicluster-data/master-2/wal/instance
I20240501 02:12:29.010227 22577 external_mini_cluster.cc:1428] Setting key cb2ca83399aa21472824eac0d5766bd5
I20240501 02:12:29.011045 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.003701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.22.12.126:35711)
0501 02:12:29.003775 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.003777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.003960 (+ 183us) server_negotiation.cc:407] Connection header received
0501 02:12:29.004399 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.004400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.004410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.004467 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.005197 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.005645 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.006231 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.006340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.008267 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.008271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.008271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.008277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.009587 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.009996 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.009997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.009998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.010033 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.010312 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.010314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.010314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.010419 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.010483 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.010836 (+ 353us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.010933 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 02:12:29.011158 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.003663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.126:35711 (local address 127.0.0.1:40330)
0501 02:12:29.003832 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:29.003858 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:29.004319 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:29.004522 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:29.004525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:29.004816 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:29.005131 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.005134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.005740 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.005741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:29.006165 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.006166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.006279 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.006790 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:29.006806 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:29.008172 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:29.009696 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.009702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.009704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.009911 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.010128 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.010129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.010130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.010229 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.010493 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:29.010499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:29.010685 (+ 186us) client_negotiation.cc:769] Sending connection context
0501 02:12:29.010814 (+ 129us) client_negotiation.cc:240] Negotiation successful
0501 02:12:29.010966 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20240501 02:12:29.011682 12977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:12:29.017623 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.011952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.22.12.125:41143)
0501 02:12:29.012014 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.012016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.012024 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:29.012045 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.012046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.012055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.012097 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.012443 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.012847 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.013356 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.013510 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.014840 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.014844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.014844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.014851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.016630 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.017023 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.017024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.017024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.017047 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.017186 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.017186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.017187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.017278 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.017331 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.017451 (+ 120us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.017524 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 02:12:29.017629 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.011889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.125:41143 (local address 127.0.0.1:50928)
0501 02:12:29.011928 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:29.011937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:29.011970 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:29.012106 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:29.012106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:29.012234 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:29.012373 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.012374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.012920 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.012921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:29.013278 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.013279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.013346 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.013654 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:29.013657 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:29.014761 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:29.016706 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.016707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.016707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.016963 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.017094 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.017095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.017096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.017139 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.017330 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:29.017331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:29.017373 (+ 42us) client_negotiation.cc:769] Sending connection context
0501 02:12:29.017424 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 02:12:29.017523 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20240501 02:12:29.018080 12977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20240501 02:12:29.018277 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.010625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.22.12.124:42963)
0501 02:12:29.010805 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.010813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.010828 (+ 15us) server_negotiation.cc:407] Connection header received
0501 02:12:29.010877 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.010880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.010895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.010965 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.011759 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.012466 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.012998 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.013188 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.015454 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.015462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.015463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.015471 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.016986 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.017277 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.017280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.017280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.017309 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.017445 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.017446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.017447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.017544 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.018057 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.018102 (+ 45us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.018175 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.124@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.126@KRBTEST.COM
I20240501 02:12:29.024338 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.018319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36704)
0501 02:12:29.018362 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:29.018372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:29.018407 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:29.018598 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:29.018599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:29.018720 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:29.018875 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.018877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.019683 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.019684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:29.019999 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.020000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.020060 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.020572 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:29.020576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:29.022099 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:29.023635 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.023636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.023636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.023757 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.023870 (+ 113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.023870 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.023871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.023896 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.024077 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:29.024077 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:29.024109 (+ 32us) client_negotiation.cc:769] Sending connection context
0501 02:12:29.024164 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 02:12:29.024241 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 02:12:29.024765 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.018433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.22.12.124:42963)
0501 02:12:29.018475 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.018477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.018483 (+ 6us) server_negotiation.cc:407] Connection header received
0501 02:12:29.018501 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.018501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.018512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.018597 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.018943 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.019610 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.020068 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.020228 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.022189 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.022192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.022192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.022198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.023566 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.023810 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.023811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.023811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.023830 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.023947 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.023948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.023948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.024030 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.024078 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.024603 (+ 525us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.024688 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240501 02:12:29.024946 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.019103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.22.12.126:35711)
0501 02:12:29.019145 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.019147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.019155 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:29.019177 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.019177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.019186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.019222 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.019822 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.020252 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.020963 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.021119 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.022887 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.022891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.022891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.022896 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.024090 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.024403 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.024404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.024404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.024420 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.024526 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.024526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.024526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.024601 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.024696 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.024777 (+ 81us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.024843 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:12:29.026687 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: Bootstrap starting.
I20240501 02:12:29.027261 12977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:29.027804 12977 log.cc:824] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:29.028501 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: No bootstrap required, opened a new log
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.12.125@KRBTEST.COM
I20240501 02:12:29.030066 12977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.030221 12977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dd698716d24b9ebc46afbff3d7d8f9, State: Initialized, Role: FOLLOWER
I20240501 02:12:29.030290 12977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.030602 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.030642 12977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:29.030707 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.031824 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.025542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.22.12.125:41143)
0501 02:12:29.025581 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.025583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.025590 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:29.025613 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.025614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.025622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.025680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.026153 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.026800 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.027299 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.027458 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.029231 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.029235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.029236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.029243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.030745 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.031173 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.031174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.031174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.031192 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.031341 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.031341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.031342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.031450 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.031572 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.031679 (+ 107us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.031746 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20240501 02:12:29.032598 12994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:29.032892 12994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:29.033360 22577 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 02:12:29.033007 12909 master_runner.cc:186] Error getting master registration for 127.22.12.126:35711: OK, instance_id { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" instance_seqno: 1714529548711829 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 02:12:29.034749 12909 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 02:12:29.034183 12801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aa564e16990d4fc0a046b7cda32f222b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40346
I20240501 02:12:29.052119 12827 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } attempt: 2
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.126@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:12:29.059546 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.052523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36712)
0501 02:12:29.052575 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:29.052588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:29.052656 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:29.052823 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:29.052824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:29.053018 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:29.053225 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.053228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.054131 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.054132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:29.054697 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.054699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.054772 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.055249 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:29.055254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:29.056782 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:29.058613 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.058615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.058615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.058797 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.058946 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.058946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.058947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.058973 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.059231 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:29.059232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:29.059277 (+ 45us) client_negotiation.cc:769] Sending connection context
0501 02:12:29.059339 (+ 62us) client_negotiation.cc:240] Negotiation successful
0501 02:12:29.059434 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 02:12:29.059561 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.052572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.22.12.124:42963)
0501 02:12:29.052659 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.052662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.052674 (+ 12us) server_negotiation.cc:407] Connection header received
0501 02:12:29.052720 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.052722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.052734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.052796 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.053318 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.054043 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.054773 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.054952 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.056865 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.056871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.056871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.056879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.058511 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.058866 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.058867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.058868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.058890 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.059022 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.059023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.059023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.059176 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.059239 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.059372 (+ 133us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.059457 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:12:29.061066 12827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b: Bootstrap starting.
I20240501 02:12:29.061728 12827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:29.061987 12827 log.cc:824] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:29.062872 12827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b: No bootstrap required, opened a new log
I20240501 02:12:29.064023 12827 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.064153 12827 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa564e16990d4fc0a046b7cda32f222b, State: Initialized, Role: FOLLOWER
I20240501 02:12:29.064213 12827 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.064507 12827 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 02:12:29.064582 12801 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 02:12:29.064636 12801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:29.065488 12801 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.065743 12801 leader_election.cc:288] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [CANDIDATE]: Term 1 election: Requested vote from peers 2768d71ddca74b8fa53aac3f8b1c7450 (127.22.12.125:41143), 15dd698716d24b9ebc46afbff3d7d8f9 (127.22.12.124:42963)
I20240501 02:12:29.064482 12997 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.066035 12997 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.066021 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa564e16990d4fc0a046b7cda32f222b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "15dd698716d24b9ebc46afbff3d7d8f9"
I20240501 02:12:29.066144 12951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:29.067050 12951 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa564e16990d4fc0a046b7cda32f222b in term 1.
I20240501 02:12:29.067366 12776 leader_election.cc:302] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dd698716d24b9ebc46afbff3d7d8f9, aa564e16990d4fc0a046b7cda32f222b; no voters:
I20240501 02:12:29.068012 12997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 02:12:29.068161 12997 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [term 1 LEADER]: Becoming Leader. State: Replica: aa564e16990d4fc0a046b7cda32f222b, State: Running, Role: LEADER
I20240501 02:12:29.068249 12997 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.068640 12997 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa564e16990d4fc0a046b7cda32f222b. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.068715 12997 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:29.069260 12759 master_runner.cc:186] Error getting master registration for 127.22.12.125:41143: OK, instance_id { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" instance_seqno: 1714529548853807 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 02:12:29.071296 12759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 02:12:29.071978 13012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:29.072031 13012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:29.072134 13018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 02:12:29.072280 13018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 02:12:29.075021 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.066986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.22.12.126:35711)
0501 02:12:29.067025 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.067027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.067339 (+ 312us) server_negotiation.cc:407] Connection header received
0501 02:12:29.067436 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.067437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.067447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.067474 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.068691 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.069340 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.070245 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.070408 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.072279 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.072282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.072283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.072289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.073920 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.074261 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.074262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.074262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.074286 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.074435 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.074436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.074436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.074572 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.074651 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.074853 (+ 202us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.074933 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:29.077683 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.067363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.22.12.125:41143)
0501 02:12:29.067401 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.067402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.068475 (+ 1073us) server_negotiation.cc:407] Connection header received
0501 02:12:29.068597 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.068598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.068609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.068649 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.069844 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.070481 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.071966 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.072164 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.074599 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.074603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.074603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.074621 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.076358 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.076813 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.076814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.076815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.076849 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.077037 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.077037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.077038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.077159 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.077229 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.077508 (+ 279us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.077593 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240501 02:12:29.079035 12951 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [term 1 FOLLOWER]: Refusing update from remote peer aa564e16990d4fc0a046b7cda32f222b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:29.079329 12997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [LEADER]: Connected to new peer: Peer: permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:29.080655 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa564e16990d4fc0a046b7cda32f222b. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.080760 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.082127 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.067805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.22.12.124:42963)
0501 02:12:29.067841 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.067843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.070758 (+ 2915us) server_negotiation.cc:407] Connection header received
0501 02:12:29.070844 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.070845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.070855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.070883 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.075982 (+ 5099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.076661 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.077330 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.077498 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.078988 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.078992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.078992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.078999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.080528 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.081185 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.081187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.081187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.081227 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.081443 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.081444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.081445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.081567 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.081656 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.081951 (+ 295us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.082036 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240501 02:12:29.082432 12997 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.082552 12997 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:29.082871 13020 mvcc.cc:205] Tried to move back new op lower bound from 7022713033023602688 to 7022713032984199168. Current Snapshot: MvccSnapshot[applied={T|T < 7022713033023602688}]
I20240501 02:12:29.083249 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.083329 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.083657 13021 mvcc.cc:205] Tried to move back new op lower bound from 7022713033023602688 to 7022713032984199168. Current Snapshot: MvccSnapshot[applied={T|T < 7022713033023602688}]
I20240501 02:12:29.085199 12902 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } attempt: 3
I20240501 02:12:29.086997 13018 catalog_manager.cc:1347] Generated new cluster ID: 8d57b4cd5f0c42589400dfd20cbb4722
I20240501 02:12:29.087141 13018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 01 02:12:29 dist-test-slave-9tkh krb5kdc[12746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714529548, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.12.125@KRBTEST.COM for kudu/127.22.12.124@KRBTEST.COM
I20240501 02:12:29.094363 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.085508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.22.12.124:42963)
0501 02:12:29.085539 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.085541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.085548 (+ 7us) server_negotiation.cc:407] Connection header received
0501 02:12:29.085600 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.085601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.085610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.085660 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.086029 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.086720 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.087344 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.087498 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.091715 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 02:12:29.091720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 02:12:29.091721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 02:12:29.091727 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0501 02:12:29.093387 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.093677 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.093678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.093679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.093702 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 02:12:29.093847 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 02:12:29.093847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 02:12:29.093848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 02:12:29.093980 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 02:12:29.094044 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:29.094189 (+ 145us) server_negotiation.cc:299] Negotiation successful
0501 02:12:29.094263 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:12:29.094363 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:29.085453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.12.124:42963 (local address 127.0.0.1:36720)
0501 02:12:29.085496 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0501 02:12:29.085507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 02:12:29.085547 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 02:12:29.085679 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 02:12:29.085680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 02:12:29.085822 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0501 02:12:29.085951 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.085953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.086787 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 02:12:29.086788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 02:12:29.087270 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 02:12:29.087273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:12:29.087334 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:29.088058 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 02:12:29.088063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0501 02:12:29.091631 (+ 3568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 02:12:29.093471 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.093472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.093473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.093611 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.093760 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 02:12:29.093761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 02:12:29.093761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0501 02:12:29.093790 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 02:12:29.094034 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 02:12:29.094035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 02:12:29.094093 (+ 58us) client_negotiation.cc:769] Sending connection context
0501 02:12:29.094157 (+ 64us) client_negotiation.cc:240] Negotiation successful
0501 02:12:29.094263 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20240501 02:12:29.096143 12902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: Bootstrap starting.
I20240501 02:12:29.096800 12902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: Neither blocks nor log segments found. Creating new log.
I20240501 02:12:29.097615 12902 log.cc:824] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: Log is configured to *not* fsync() on all Append() calls
I20240501 02:12:29.098670 12902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: No bootstrap required, opened a new log
I20240501 02:12:29.100554 12902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.100689 12902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2768d71ddca74b8fa53aac3f8b1c7450, State: Initialized, Role: FOLLOWER
I20240501 02:12:29.100761 12902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } }
I20240501 02:12:29.101078 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.101159 12875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [term 0 FOLLOWER]: Advancing to term 1
I20240501 02:12:29.101179 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa564e16990d4fc0a046b7cda32f222b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2768d71ddca74b8fa53aac3f8b1c7450"
I20240501 02:12:29.101087 12902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 02:12:29.101953 12855 tablet.cc:2369] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 02:12:29.102249 12875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [term 1 FOLLOWER]: Refusing update from remote peer aa564e16990d4fc0a046b7cda32f222b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 02:12:29.102872 13016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 02:12:29.103524 12876 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa564e16990d4fc0a046b7cda32f222b in term 1.
I20240501 02:12:29.104020 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.104812 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa564e16990d4fc0a046b7cda32f222b. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.104914 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.104949 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.104984 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 02:12:29.105423 13018 catalog_manager.cc:1370] Generated new certificate authority record
W20240501 02:12:29.105737 13038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 02:12:29.105789 13038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 02:12:29.106171 13018 catalog_manager.cc:1504] Loading token signing keys...
I20240501 02:12:29.107992 12997 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa564e16990d4fc0a046b7cda32f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa564e16990d4fc0a046b7cda32f222b" member_type: VOTER last_known_addr { host: "127.22.12.126" port: 35711 } } peers { permanent_uuid: "2768d71ddca74b8fa53aac3f8b1c7450" member_type: VOTER last_known_addr { host: "127.22.12.125" port: 41143 } } peers { permanent_uuid: "15dd698716d24b9ebc46afbff3d7d8f9" member_type: VOTER last_known_addr { host: "127.22.12.124" port: 42963 } } }
I20240501 02:12:29.108091 12997 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b [sys.catalog]: This master's current role is: LEADER
I20240501 02:12:29.117524 13018 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aa564e16990d4fc0a046b7cda32f222b: Generated new TSK 0
I20240501 02:12:29.117751 13018 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 02:12:29.145768 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.144288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.22.12.125:41143)
0501 02:12:29.144335 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.144337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.145060 (+ 723us) server_negotiation.cc:407] Connection header received
0501 02:12:29.145172 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.145173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.145185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.145213 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.145698 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50952: BlockingRecv error: recv got EOF from 127.0.0.1:50952 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240501 02:12:29.146585 12981 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.144738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.22.12.124:42963)
0501 02:12:29.144780 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.144782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.146029 (+ 1247us) server_negotiation.cc:407] Connection header received
0501 02:12:29.146126 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.146127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.146138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.146164 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.146509 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36724: BlockingRecv error: recv got EOF from 127.0.0.1:36724 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240501 02:12:29.147662 12831 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.146957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.22.12.126:35711)
0501 02:12:29.147012 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.147014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.147125 (+ 111us) server_negotiation.cc:407] Connection header received
0501 02:12:29.147222 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.147223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.147233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.147261 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.147602 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40376: BlockingRecv error: recv got EOF from 127.0.0.1:40376 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20240501 02:12:29.150147 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.149390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.22.12.125:41143)
0501 02:12:29.149441 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.149443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.149621 (+ 178us) server_negotiation.cc:407] Connection header received
0501 02:12:29.149733 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.149734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.149751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.149778 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.150096 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50960: BlockingRecv error: recv got EOF from 127.0.0.1:50960 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20240501 02:12:29.153110 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.152381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.22.12.125:41143)
0501 02:12:29.152423 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.152425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.152655 (+ 230us) server_negotiation.cc:407] Connection header received
0501 02:12:29.152737 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.152738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.152747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.152771 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.153063 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50966: BlockingRecv error: recv got EOF from 127.0.0.1:50966 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20240501 02:12:29.156950 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.156169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.22.12.125:41143)
0501 02:12:29.156202 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.156204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.156454 (+ 250us) server_negotiation.cc:407] Connection header received
0501 02:12:29.156550 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.156551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.156561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.156587 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.156900 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50968: BlockingRecv error: recv got EOF from 127.0.0.1:50968 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20240501 02:12:29.163100 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.162141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.22.12.125:41143)
0501 02:12:29.162183 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.162185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.162489 (+ 304us) server_negotiation.cc:407] Connection header received
0501 02:12:29.162622 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.162623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.162634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.162662 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.163042 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50980: BlockingRecv error: recv got EOF from 127.0.0.1:50980 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20240501 02:12:29.173470 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.172466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.22.12.125:41143)
0501 02:12:29.172512 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.172514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.172839 (+ 325us) server_negotiation.cc:407] Connection header received
0501 02:12:29.172950 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.172951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.172963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.172992 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.173393 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50988: BlockingRecv error: recv got EOF from 127.0.0.1:50988 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240501 02:12:29.200326 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.199359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.22.12.125:41143)
0501 02:12:29.199416 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.199418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.199710 (+ 292us) server_negotiation.cc:407] Connection header received
0501 02:12:29.199815 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.199816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.199829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.199856 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.200260 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51004: BlockingRecv error: recv got EOF from 127.0.0.1:51004 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240501 02:12:29.234311 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.233585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.22.12.125:41143)
0501 02:12:29.233628 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.233630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.233754 (+ 124us) server_negotiation.cc:407] Connection header received
0501 02:12:29.233820 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.233821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.233833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.233881 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.234247 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51016: BlockingRecv error: recv got EOF from 127.0.0.1:51016 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240501 02:12:29.300354 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.299542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.22.12.125:41143)
0501 02:12:29.299591 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.299593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.299760 (+ 167us) server_negotiation.cc:407] Connection header received
0501 02:12:29.299820 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.299821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.299834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.299903 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.300291 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51022: BlockingRecv error: recv got EOF from 127.0.0.1:51022 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20240501 02:12:29.430121 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.429405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.22.12.125:41143)
0501 02:12:29.429494 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.429496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.429554 (+ 58us) server_negotiation.cc:407] Connection header received
0501 02:12:29.429610 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.429611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.429622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.429675 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.430024 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51026: BlockingRecv error: recv got EOF from 127.0.0.1:51026 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20240501 02:12:29.688032 12906 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 02:12:29.687360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.22.12.125:41143)
0501 02:12:29.687429 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:29.687431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:29.687442 (+ 11us) server_negotiation.cc:407] Connection header received
0501 02:12:29.687501 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:29.687501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:29.687512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:29.687559 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 02:12:29.687960 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51036: BlockingRecv error: recv got EOF from 127.0.0.1:51036 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 02:12:30.033345 12994 catalog_manager.cc:1259] Loaded cluster ID: 8d57b4cd5f0c42589400dfd20cbb4722
I20240501 02:12:30.033430 12994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: loading cluster ID for follower catalog manager: success
I20240501 02:12:30.034335 12994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: acquiring CA information for follower catalog manager: success
I20240501 02:12:30.034670 12994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15dd698716d24b9ebc46afbff3d7d8f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:12:30.106225 13038 catalog_manager.cc:1259] Loaded cluster ID: 8d57b4cd5f0c42589400dfd20cbb4722
I20240501 02:12:30.106312 13038 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: loading cluster ID for follower catalog manager: success
I20240501 02:12:30.107236 13038 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: acquiring CA information for follower catalog manager: success
I20240501 02:12:30.107537 13038 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2768d71ddca74b8fa53aac3f8b1c7450: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 02:12:30.203682 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:30.201212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.22.12.125:41143)
0501 02:12:30.201313 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:30.201315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:30.201378 (+ 63us) server_negotiation.cc:407] Connection header received
0501 02:12:30.201442 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:30.201443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:30.201484 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:30.201569 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:30.201961 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.202463 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:30.203144 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.203291 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:30.203335 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:30.203429 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:30.203499 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:30.203567 (+ 68us) server_negotiation.cc:299] Negotiation successful
0501 02:12:30.203595 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":55,"threads_started":1}
I20240501 02:12:30.207412 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:30.205426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.22.12.124:42963)
0501 02:12:30.205526 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:30.205528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:30.205537 (+ 9us) server_negotiation.cc:407] Connection header received
0501 02:12:30.205601 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:30.205602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:30.205627 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:30.205699 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:30.205891 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.206386 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:30.206913 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.207026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:30.207063 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:30.207167 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:30.207251 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:30.207305 (+ 54us) server_negotiation.cc:299] Negotiation successful
0501 02:12:30.207337 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20240501 02:12:30.211316 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:30.209090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.22.12.125:41143)
0501 02:12:30.209135 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:30.209137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:30.209283 (+ 146us) server_negotiation.cc:407] Connection header received
0501 02:12:30.209339 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:30.209340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:30.209355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:30.209404 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:30.209670 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.210169 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:30.210868 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.211051 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:30.211074 (+ 23us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:30.211105 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:30.211161 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:30.211214 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 02:12:30.211235 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 02:12:30.211551 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:30.209187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.22.12.124:42963)
0501 02:12:30.209218 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:30.209220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:30.209233 (+ 13us) server_negotiation.cc:407] Connection header received
0501 02:12:30.209325 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:30.209326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:30.209339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:30.209365 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:30.209698 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.210372 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:30.211097 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.211262 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:30.211283 (+ 21us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:30.211308 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:30.211359 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:30.211459 (+ 100us) server_negotiation.cc:299] Negotiation successful
0501 02:12:30.211491 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240501 02:12:30.211910 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 02:12:30.209108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.22.12.126:35711)
0501 02:12:30.209335 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0501 02:12:30.209337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 02:12:30.209345 (+ 8us) server_negotiation.cc:407] Connection header received
0501 02:12:30.209389 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 02:12:30.209390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 02:12:30.209425 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 02:12:30.209523 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 02:12:30.209790 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.210569 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:12:30.211225 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 02:12:30.211408 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:12:30.211476 (+ 68us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:12:30.211552 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:12:30.211652 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 02:12:30.211667 (+ 15us) server_negotiation.cc:299] Negotiation successful
0501 02:12:30.211836 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":44,"threads_started":1}
I20240501 02:12:30.213819 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12759
I20240501 02:12:30.219015 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12834
I20240501 02:12:30.223675 22577 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKBEtG/build/release/bin/kudu with pid 12909
2024-05-01T02:12:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2641 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5265 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (300817 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
1 FAILED TEST